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Abstract 

In  order  to  meet  the  challenges  of  a  reduced  work  force 
and  the  changing  roles  and/or  missions  of  the  Air  Force  in 
particular,  it  is  imperative  that  all  measures  available  be 
taken  to  effectively  utilize  current  resources.  Currently 
there  is  a  Cost  and  Operational  Effectiveness  Analysis 
(COEA)  used  to  assist  in  the  decision-making  of  every  phase 
of  the  acquisition  process.  The  Office  of  the  Assistant 
Secretary  of  Defense  (OASD)  has  mandated  that  COEAs  are  to 
be  an  integral  part  of  the  acquisition  process. 

The  COEA  information  gathering  or  sharing  process  is 
not  well  defined.  Areas  within  the  COEA  process  affected 
are  the  coordination  of  common  elements  of  information 
required,  the  data  collection,  and  the  generation  of 
possible  solutions.  The  problem  addressed  by  this  research 
is  how  to  improve  the  COEA  information  sharing  process  for 
data  used  to  produce  analyses  for  an  organization.  This 
improved  process  should  result  in  a  reduction  of  the  time 
spent  in  continual  meetings  and  conferences  resolving 
conflicts  within  the  process  areas. 

The  result  of  our  research  indicates  that  the  different 
processes  within  the  COEIA  information  process  could  be 
organized  within  a  knowledge-based  system  (KBS)  for 
improving  the  sharing  of  information  and  the  overall 
efficiency  of  the  process. 
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INFORMATION  SHARING 


VfITHIN  THE 
COEA  PROCESS 


L _ Introduction 


General  Issues 

In  the  next  several  years,  the  Department  of  Defense 
(DoD)  will  be  downsizing  both  personnel  and  material 
resources.  The  Commander  of  the  Air  Force  Material  Command 
(AFMC),  General  Ronald  W.  Yates,  noted  in  a  briefing  before 
Congress , 

We  were  able  to  absorb  the  first  round  of  cuts  by 
such  belt-tightening  measures  as  cancelling  vacant 
authorizations,  placing  surplus  employees  on  other 
valid  positions  and  attrition.  Unfortunately,  due 
to  the  magnitude  of  the  future  overstrength  posture, 
AFMC  could  no  longer  resolve  the  problem  for  fiscal 
1994  and  beyond.  Additional  reductions,  in  excess 
of  projected  attrition,  were  projected  for  fiscal 
1994-95.  (1993:4) 

In  order  to  meet  the  challenges  of  a  reduced  work  force 
and  the  changing  roles  and/or  missions  of  the  Air  Force  in 
particular,  it  is  imperative  that  all  measures  available  be 
taken  to  effectively  utilize  current  resources. 


BagKgrownd 

Subject  matter  experts  expect,  with  the  downsizing,  the 
Air  Force  will  be  modifying  more  existing  aircraft  as 
opposed  to  designing  and  building  new  aircraft  (Cronk, 


1 


Figure  1.  DttVttlopm«nt  Plaiuiing  Dir*ctorat« 

1993) .  To  help  meet  the  challenges  of  supporting  these 
increased  modifications,  the  Studies  and  Analyses  Division 
(XRE)  of  the  Aeronautical  Systems  Center  (ASC)  Directorate 
of  Development  Planning  (XR)  is  tasked  to  produce  one  stop 
studies  and  analyses.  Figure  1  shows  the  organizational 
structure  for  XR.  "One  stop"  means  the  customer  only 
interacts  with  one  group  to  gain  their  desired  information. 
XRE's  primary  customers  are  headquarters  staffs  within  Air 
Force  Material  Command  (AFMC),  Air  Force  operations 
commands.  Systems  Programs  Offices  within  ASC,  and  AFMC 
laboratories . 

XRE's  Campaign  and  Analysis  Branch  (XREC)  provides 
feedback  on  a  simulated  series  of  military  operations 
forming  a  distinct  phase  of  a  war  (i.e.  campaign) .  These 
simulated  campaigns  focus  on  the  entire  scope  of  land  and 
air  area  that  may  become  involved  directly  in  military 
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engagements  (i.e.  theater-level) .  These  simulations  produce 
the  perspectives  necessary  to  evaluate  modifications  to 
existing  systems  and  potential  acquisitions  of  future 
systems.  The  Theater  Analysis  and  Resource  Analysis 
sections  of  XREC  have  the  primary  responsibility  for  two  of 
the  simulation  models  currently  used  to  provide  data  for 
studies  and  analyses . 

The  Theater  Analysis  Section  conducts  theater-level 
campaign  analysis  through  the  use  of  the  simulation  model 
TAG  THUNDER.  TAG  THUNDER  is  a  two-sided  theater-level  war 
simulation  program  that  models  air  and  ground  combat  and 
logistics  scenarios.  The  scenarios  encompass  such  varied 
areas  as  force  structure,  terrain,  and  weapons  systems  as 
described  by  the  user-supplied  data.  This  simulation  allows 
the  analyst  to  study  the  effects  of  the  changes  in  plans, 
tactics,  force  structures,  and  weapons  systems  at  the 
theater- level . 

Focusing  on  the  general  areas  of  supportability  and 
affordability  of  the  modification  or  acquisition  proposed, 
the  Resource  Analysis  Section  cpaantifies  the  resources 
required  to  accomplish  the  various  objectives  for  existing 
and  future  systems  and  subsystems.  The  Logistics  Gomposite 
Model  (LGOM)  creates  a  representation  of  the  work  flow  found 
in  a  maintenance  organization  and  produces  data  that  is  then 
passed  to  the  cost  analysis  section  of  XRE  for 
identification  of  the  optimal  blend  of  resources  to  support 
a  weapon  system  under  peace-time  and  war-time  operating 
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conditions .  More  recently,  LOOM  has  been  paired  with 
comparability  analysis  techniques  to  produce  a  baseline 
configuration  for  new  or  modified  systems .  This  comparative 
analysis  feature  was  added  to  LCOM  to  anticipate  the 
"problem"  of  the  non-availability  of  in-house  logistics 
support  data  for  comparison.  The  analyst  will  then  look  at 
other  sources  for  comparative  data. 

Whether  a  new  or  modified  system  is  envisioned,  each 
major  command  (MAJCOM)  is  responsible  to  produce  a  Cost  and 
Operational  Effectiveness  Analysis  (COEA) .  The  COEA  is 
required  by  the  Office  of  the  Assistant  Secretary  of  Defense 
(OASD)  to  provide  analytical  rationale,  facilitate 
discussion,  and  establish  audit  trails  for  milestone 
decisions  regarding  the  acquisition  of  the  new  or  modified 
system,  i^pendix  G  explains  the  five  different  types  of 
COEAs  that  could  possibly  be  required  during  the  acquisition 
lifecycle.  The  results  from  each  type  of  COEIA  are  critical 
for  selecting  the  best  possible  system  to  meet  the  customers 
requirements  at  that  given  phase.  The  Development  Planning 
Directorate  of  the  Aeronautical  Systems  Center  (ASC/XR)  is 
tasked  with  approximately  20  COEAs  per  year. 

These  COEAs  are  aircraft-based  studies  based  on 
deficiencies,  opportunities,  or  obsolescence  issues  as 
specified  by  the  customer.  COEA  analyses  require  a 
tremendous  amount  of  manual  manipulation  of  data  to  produce 
the  final  reports  and  supporting  documentation.  For  the 


4 


average  COEA  effort,  XRE  estimates  each  COEA  will  take  at 
least  120  manhours. 

With  the  downsizing  of  both  personnel  and  material, 
the  number  of  COEAs  due  to  modifications  will  rise.  This 
means  that  the  various  models  utilized  by  the  analysts  in 
XRE  to  predict  weapon  system  capabilities,  survivability, 
and  mission  effectiveness  will  be  used  to  an  even  greater 
extent.  The  standard  operating  procedures  in  place  for  all 
tasks  require  each  group  of  analysts  to  enter  individual 
input  parameters  to  their  simulation  models.  Mr.  Richard 
Cronk,  LCOM  Group  Leader,  says  one  of  the  severest  limiting 
factors  within  the  XRE  environment  is  no  coordination  among 
the  various  simulation  groups  during  the  development  of  the 
solution.  At  present  there  is  no  way  of  knowing  whether 
each  group  has  made  its  predictions  using  all  the  same 
criteria  and  assumptions . 

Only  after  each  model  selected  to  contribute  data  to 
the  COEA  has  been  run  is  any  coordination  begun.  This 
coordination  is  currently  done  manually  during  the 
integration  of  the  selected  model  outputs.  The  coordination 
approach  utilizes  the  stubby  pencil  method  while  trying  to 
analyze  what  criteria  and/or  assumptions  were  used  by  each 
group  through  the  time  consuming  method  of  conferences. 

Mr.  John  M.  Griffin,  Director  of  ASC's  Development 
Planning  Directorate  (ASC/XR),  stated  in  his  letter  dated  23 
December  1992  that  the  "new  interest  in  COEA  support  and 
organic  modeling  and  simulations  capability  demands  that  ASC 
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stays  vibrant"  (Griffin,  1992:  1).  To  remain  "vibrant"  in 
this  era  of  downsizing  means  harnessing  every  potential 
manpower-saving  tool  within  the  Air  Force's  technological 
grasp. 

Mr.  Griffin  further  stated  in  the  same  letter  that 
Studies  and  Analysis  (XRE)  "has  been  an  ASC  mainstay  for 
years.  I  believe  our  XRE  capability  will  definitely  be 
needed  as  the  Air  Force  faces  a  future  of  hard  decisions 
driven  by  reduced  resources"  (Griffin,  1992:1).  XRE 
currently  is  involved  in  at  least  20  COEAs  a  year.  The 
senior  leadership  within  XR  fully  expects  the  number  of 
COEAs  to  increase  significantly,  especially  with  a 
downsizing  trend.  Due  to  the  economic  realities  that  it  is 
cheaper  to  modify  existing  airframes  for  new  missions  than 
to  build  new  specific  mission-based  airframes  as  the  Air 
Force  and  other  services  have  done  in  the  past,  the  number 
of  COEAs  will  increase.  Such  an  increase  will  strain  (if 
not  overload)  the  current  capabilities  within  XRE  to  produce 
the  foreseen  number  of  COEAs.  One  answer  is  to  produce 
tools  that  will  expand  the  current  capabilities  without 
increasing  the  manpower  base  currently  in  place. 

Specific  Problem 

The  problem  addressed  by  this  research  is  how  to 
improve  the  COEA  information  sharing  process  for  data  used 
to  produce  analyses  for  an  organization. 
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Problem  Statement 


Within  Aeronautical  Systems  Center's  Studies  and 
Analysis  Division  (ASC/XRE) ,  the  Campaign  Analysis  Branch 
(XREC)  is  a  process-based  organization  that  flows  each 
product  except  the  Cost  and  Operational  Effectiveness 
Analysis  (COEA)  through  a  standardized  process .  The  COEA  is 
a  product  that  has  been  mandated  by  the  Office  of  the 
Assistant  Secretary  of  Defense  (OASD)  as  an  essential  part 
of  the  weapons  systems  acquisition  process.  An  information 
system  to  perform  the  information  sharing  necessary  for  each 
COEA  needs  to  be  developed.  By  definition,  an  information 
system  is  "an  open  purposive  system  that  produces 
information  using  the  input/process/output  cycle.  The 
minimal  information  system  consists  of  people,  procedures, 
and  data"  (Kroenke,  1992:782) . 

Objectives 

The  objectives  for  organizing  the  COEA  process  within 
an  information  system  framework  which  will  prove  the  concept 
of  information  sharing  are  as  follows: 

1 .  Identify  the  information  that  is  commonly  used  by 
the  two  models. 

2.  Develop  the  specific  form  the  information  will  be 
in  when  passed  to  the  two  models. 

3 .  Identify  the  assumptions  made  by  each  model  as  to 
the  meaning  of  the  information  and  how  it  will  be  used 
in  the  model . 
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4 .  Identify  the  various  integrated  scenarios  that 
could  derive  the  needed  data. 

5.  Develop  a  prototype  information  system. 

Limitations  on  the  Scope  of  the  Research 

Since  the  objective  of  the  research  is  to  prove  the 
concept  of  information  sharing  through  an  information 
system,  the  main  limitation  is  the  selection  of  appropriate 
information  and  scenarios  that  will  be  a  representative 
sample  of  the  overall  problem.  The  simulation  models 
selected  for  this  study  are  integral  in  almost  every  COEA. 
Another  limitation  is  how  closely  the  structured  and 
unstructured  knowledge  of  the  experts  can  be  modeled  in  the 
COEA  infoanaation  sharing  system  that  will  function  as  the 
overall  data  information  source.  A  third  limiting  factor  is 
the  inherent  limitation  of  all  expert-system-building  tools, 
their  inability  to  directly  interact  with  the  domain  expert, 
thus  limiting  their  ability  to  acquire  knowledge. 

Justification  of  the  Research 

The  justification  for  the  desired  capability  of  sharing 
the  information  to  be  used  by  each  of  these  models  is 
two-fold.  First,  there  is  the  need  to  support  the  Integrated 
Product  Development  program  within  ASC.  There  is  a  current 
initiative  within  ASC  from  the  commander  that  states  all 
projects  will  comply  with  an  integrated  product  development 
cycle. 
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I  see  this  operational  initiation  as  the  culmination 
of  the  management  direction  coming  out  of  the 
commander's  off-sites  to  effect  a  world  class 
development  planning  capability .  XR' s  role  in  the 
commander's  vision  is  to  provide  a  highly 
professional  study  and  analysis  function  including 
strong  emphasis  on  the  customer  interface  through 
planning  and  roadmapping,  an  aggressive  search  for 
opportunities  to  exploit  new  concepts  for  making 
quantum  improvements  in  Air  Force  war  fighting 
capability,  and  providing  investment  strategy 
guidance  to  the  laboratories  for  support  and  timely 
transition  of  their  science  and  technology  programs . 
(Boyd,  1992) 

Last,  in  support  of  the  continuing  downsizing  of  DoO, 
ASC/XR  proposed  a  reduction  of  30  per  cent  of  the  combined 
manpower  within  ASC/XR.  To  support  this  reduction, 
improving  the  current  processes  through  the  use  of  a 
standardized  information  source  will  help  reduce  the  effect 
of  this  projected  cutback. 


The  downsizing  of  DoD  is  forcing  several  initiatives  to 
take  place  to  enable  better  utilization  of  the  remaining 
personnel  and  material  resources.  Automating  procedures 
which  are  currently  done  manually  is  one  such  way  that 
better  utilization  can  be  accomplished.  The  sharing  of 
common  information  to  be  used  as  guidance  for  producing  the 
parameters  for  use  within  the  models  in  an  automated  manner 
is  one  such  example  where  manpower  can  be  better  utilized. 

Definition  of  Terms 

Terms  used  throughout  the  thesis  are  defined  in 
Appendix  A. 
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Overview 


Chapter  2  contains  a  review  of  the  literature  of 
successful  applications  of  information  sharing  as  a 
solution.  Chapter  3  provides  the  methodology  of  the 
development  of  an  information  system  solution  for  the 
successful  sharing  of  information  to  be  used  as  guidance  for 
parameter  generation  for  LCOM  and  TAC  THUNDER.  The  results 
of  applying  the  methodology  outlined  in  Chapter  3  will  be 
the  focus  of  Chapter  4 .  The  conclusions  of  the  study  and 
recommendations  for  any  further  study  or  action  to  be  taken 
by  XRE  will  be  found  in  Chapter  5. 
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II.  Literature  Review 


Introduction 

The  purpose  of  this  chapter  would  normally  be  to  review 
the  literature  which  discussed  actual  solutions  other 
organizations  (companies,  etc.)  have  implemented  in  similar 
situations.  Part  of  the  problem  with  finding  any  relevant 
information  is  wading  through  the  hyperbole  and  getting  dotm 
to  the  cold,  hard,  facts  of  what  exactly  is  data  sharing. 
Data  sharing  occurs  "when  two  general  conditions  are 
satisfied:  (1)  they  (data)  are  used  by  organizational 
members  and  (2)  they  can  be  linked  to  organizational 
effectiveness"  (Wyse  and  Higgins,  1993:34) . 

Review  Findings 

The  researchers  conducted  extensive  searches  through 
the  Air  Force  Institute  of  Technology  (AFIT)  Library,  the 
Defense  Technical  Information  Center  (DTIC) ,  the  Dialog 
Service  electronic  library  and  the  National  Aeronautics  and 
Space  Administration  (NASA)  research  library.  Although  the 
searches  produced  several  hundred  citations,  the  majority  of 
the  citations  were  not  relevant  to  the  research  effort.  The 
keywords  used  in  each  of  the  searches  were:  data  sharing, 
data  integration,  data  management,  database  management, 
relational  database  management  system,  data  simulation. 
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Information  transfer,  technology  transfer,  information  flow, 
and  information  management. 

The  only  citations  that  appeared  relevant  to  the 
research  effort  dealt  with  integration  or  sharing  of  the 
data  shared  by  two  or  more  databases  (Kamel  and  Zviran, 

1991)  (Walker,  1990).  After  further  research  along  this 
line  in  LCOM  and  TAG  THUNDER,  the  researchers  determined 
with  Mr.  Cronk's  and  Captain  McCormick’s  help  that  the 
information  to  be  shared  was  not  of  the  form  to  be  stored 
and  retrieved  from  a  database  format. 

There  are  references  to  the  theory  of  the  use  of 
knowledge-based  systems  to  solve  the  problems  of  sharing 
information.  These  references  were  found  within  trade 
journals,  textbooks,  and  popular  literature.  But  no  actual 
applications  were  mentioned  in  any  of  the  references. 

It  is  the  researcher's  speculation  that  the  lack  of 
published  examples  of  successful  applications  is  due  in  part 
to  the  competitiveness  of  business.  In  informal  discussions 
with  a  senior  systems  engineer  at  a  leading-edge  information 
technology  business,  he  supported  the  researchers' 
conclusions  by  stating  information  sharing  to  this  degree 
would  be  the  competitive  edge. 

It  is  still  the  researcher's  contention  that  using 
knowledge-based  systems  is  another  avenue  for  the  sharing  of 
information.  Such  an  application  has  great  potential  to 
increase  an  organization's  effectiveness  and  efficiency 
through  interoperability. 


12 


Summary 

The  researchers,  after  reviewing  several  hundred 
citations,  found  no  documented  successful  applications  of  an 
information  system  that  handled  the  same  type  of  information 
as  needed  by  LOOM  and  TAC  THUNDER.  Therefore  the  research 
is  the  first  to  be  documented  in  the  area  of  using  a 
knowledge-based  information  system  to  share  information. 
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III.  Methodology 

Overview 

This  chapter  will  provide  a  discussion  of  the 
methodology  to  be  used  in  this  thesis.  The  basic  form  of 
the  methodology  being  utilized  is  documented  by  Mockler  and 
Dologite  in  their  1992  book  Knowledae-Based  Systems .  An 
Introduction  to  Expert  Systems.  The  chapter  will  culminate 
with  a  table  that  summarizes  the  steps  necessary  to  build  a 
knowledge-based  system. 

Justification  of  the  use  of  a  Knowledge  Based  System 

With  the  current  trend  of  downsizing,  human  expertise 
will  become  even  more  scarce  and  needed  in  a  variety  of 
locations  and  projects .  A  tool  is  required  that  will  allow 
anyone  with  a  working  knowledge  of  the  COEA  process  to  begin 
gathering  information  without  immediate  access  to  the  human 
experts . 

The  needs  of  XREC  fall  under  the  planning  category  for 
knowledge  based  systems  (KBSs)  as  found  in  the  Mockler  and 
Dologite  text  on  p.  17.  The  KBS  that  XREC  needs  would  play 
an  integral  part  in  the  military  planning  of  how  to  wisely 
spend  the  ever-shrinking  defense  dollars  the  Department  of 
Defense  has  been  appropriated. 
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The  benefits  for  applying  a  KBS  to  this  process  are  as 


follows : 

The  KBS  provides  for  a  consistent  level  of  service 
to  be  delivered,  regardless  of  who  is  on  duty. 

The  KBS  makes  possible  decision-making  by  personnel 
who  were  previously  unauthorized  to  make  decisions. 

The  KBS  ensures  that  decisions  are  always  made  using 
the  same  set  of  criteria.  When  the  COEA  is  then 
integrated,  this  KBS  tool  will  help  in  resolving  any 
conflicting  findings  between  the  simulation  models. 

The  KBS  can  be  used  to  train  personnel,  which  frees 
more  experienced  staff  for  other  duties. 

The  KBS  can  be  replicated  and  used  wherever  COEAs 
are  processed,  assuring  the  organization  of  a 
consistent  level  of  service. 

The  KBS  can  easily  be  changed  to  reflect  new  or 
revised  process  regulations  and  then  be  quickly 
replicated  and  distributed  to  implement  the  change 
uniformly  throughout  the  organization  without 
incurring  personnel  retraining  expenses. 

(Mockler  and  Dologite; 102-103) 


Waterman  (1986:186-199)  uses  three  chapters  in  his  book 
to  discuss  in  depth  a  variety  of  pitfalls  every  developer 
might  encounter  within  the  areas  of  expert  system  planning 
and  development.  The  areas  listed  below  are  the  more  common 
ones  to  look  for  major  pitfalls  within  any  expert  system 
development  effort.  Beside  each  listed  pitfall  are  the 
steps  the  researchers  took  to  avoid  each  area  that  applied 
to  their  development  effort. 
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1.  Choosing  an  Appropriate  Problem.  This  problem 
was  dealt  with  by  a  series  of  interactive 
assignments  within  the  Artificial  Intelligence  class 
taught  by  Captain  Michael  Shouhat.  By  turning  in  a 
topic  outline,  a  specific  problem  statement,  and  a 
midterm  project  (that  included  all  the  KBS 
development  steps  except  entering  the  KBS  into  the 
computer) ,  the  researchers  ensured  the  problem  was 
not  only  appropriate  but  properly  scoped  in  size. 

2.  Choosing  the  Expert  System  Building  Tool.  The 
researchers  chose  EXSYS  Professional  since  this 
tool's  capabilities  matched  the  problem  domain 
characteristics  as  found  in  the  initial  knowledge 
acquisition. 

3.  Choosing  the  Domain  Expert.  This  was  not  a 
problem  for  the  researchers  since  both  LCOM  and  TAC 
THUNDER  provided  the  best  experts  available. 

4.  Interacting  with  the  Domain  Expert.  This 
problem  was  averted  by  having  dealt  with  these 
experts  for  months  during  the  initial  thesis 
research  and  problem  definition. 

5.  System  Implementation.  Most  of  the  pitfalls 
within  this  area  were  handled  by  testing  the  rules 
as  they  were  developed  through  a  series  of 
interviews.  Any  other  problems  were  eliminated  by 
using  EXSYS  Professional  and  many  of  its  validation 
and  inference-checking  capabilities. 
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6.  System  Testing  and  Evaluation.  The  pitfalls 
usually  associated  with  this  area  were  avoided  by 
using  the  appropriate  planning  techniques  as 
specified  by  Captain  Shoukat.  Other  design  flaws 
that  could  appear  during  this  stage  of  KBS 
development  were  handled  through  the  use  of 
ergonomic  development  techniques  as  outlined  in 
Human  Factors  in  Engineering  and  Design  by  Sanders 
and  McCormick. 

Development  of  Knowledae-Based  System 

After  careful  review  of  the  steps  for  the  development 
of  a  KBS  as  found  in  several  sources  (Hayes-Roth,  1983:139) 
(Waterman,  1986:137)  (Mockler  and  Dologite,  1992:46)  (Irwin 
1991:3-1)  (Nelson,  1991:41),  the  researchers  have  concluded 
that  KBS  development  can  be  broken  down  into  five  phases: 
Project  Planning,  Analysis,  Transformation,  Implementation, 
and  Testing.  Figure  2  demonstrates  the  recursive  nature  of 
the  five  phases. 

A.  Project  Planning  Phase 

The  purpose  of  the  Project  Planning  Phase  is  to 
determine  the  area  of  study,  properly  scope  the  area  of 
study,  and  to  determine  whether  or  not  to  proceed  with  the 
project.  This  process  studies  the  business  need,  the 
feasibility  of  the  project,  and  the  cost/benefit  comparison 
of  the  KBS.  In  the  pursuit  of  the  answers  to  these  steps, 
the  developer  is  involved  in  the  initial  knowledge 
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REQUIREMENTS  DECISIONS  STRUCTURE  RULES 


Figure  2.  Phases  of  KBS  Developeient 

acquisition  of  the  project.  The  culminating  product  of  this 
phase  is  to  determine  the  level  of  risk  for  the  initial 
prototype . 


1.  Selection  of  the  Project. 

One  of  the  many  reasons  for  building  a 
knowledge-based  expert  system  is  that  the  human  expertise  of 
both  Mr.  Richard  Cronk,  Group  Leader  of  the  Logistics 
Composite  Model  (LCOM) ,  and  Captain  Dave  McCormick, 
Operations  Research  Analyst  for  TAC  THUNDER,  is  needed  in  a 
variety  of  locations.  At  present,  neither  of  these  group 
leaders  are  located  in  the  same  building  which  makes 
interaction  on  many  projects  difficult  at  best.  Since  the 
Office  of  the  Assistant  Secretary  of  Defense  (OASD)  requires 
Cost  and  Operational  Effectiveness  Analyses  (COEAs)  for  the 
acquisition  of  many  DoD  acquisition  categories,  the  COEA  has 
become  an  "essential"  part  of  the  acquisition  process. 
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According  to  AFSC  Pamphlet  (AFSCP)  173-1,  "COEAs  are 
comparative  analyses  of  the  costs  and  operational 
effectiveness  of  alternative  solutions  intended  to  satisfy 
an  established  mission  need"  (p.  2) .  Out  of  the  six  COEAs 
currently  underway,  LOOM  is  involved  in  some  way  in  all  six 
while  TAG  THUNDER  is  involved  in  five  out  of  the  six. 

2 .  Definition 

The  prototype  KBS  will  integrate  data  from 
several  of  the  necessary  fields  within  a  specified  LCOM 
scenario.  In  order  to  help  answer  a  number  of  the  areas  of 
concern  within  the  COEA,  an  analysis  will  be  performed  to 
select  the  appropriate  data  from  among  the  various  fields  of 
possible  data.  Sources  for  this  analysis  will  include 
written  guidelines  for  the  COEA  process  and  interviews  with 
Mr  Cronk  that  will  identify  the  decisions  to  be  modeled  and 
determine  which  specific  areas  always  have  an  overall 
importance  to  the  final  COEA  answer. 

3 .  Preliminary  Screening 

According  to  Mockler  and  Dologite  (1992:47- 
48) ,  several  questions  need  to  be  answered  as  part  of  the 
preliminary  screening  process.  The  purpose  of  these 
questions  is  to  determine  the  feasibility  and 
appropriateness  of  the  area  under  study.  These  include: 

a.  Do  recognized  experts  exist? 

b.  Can  the  experts  do  the  task  better  than 
amateurs  and  can  their  skills  be  taught  to 
others? 
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c.  Do  different  experts  agree  on  the 
solutions? 


d.  Does  the  task  require  reason  and  informed 
judgments,  as  opposed  to  mere  common  sense? 

e.  Is  the  task  well  understood? 

f.  Can  the  experts  articulate  their  methods? 

g.  Is  the  task  of  manageable  size? 

h.  Are  typical  example  cases  or  situations 
readily  available? 

The  specific  answers  to  the  above  questions  are  found 
in  Chapter  4 . 

4 .  Estimating  the  level  of  risk 

Several  areas  of  concern  that  need  to  be 
addressed  when  determining  the  level  of  risk  are  found 
within  Mockler  and  Dologite  (1992:53). 


a.  Knowledge  Area  Complexity  can  be  Simple, 
Moderate,  or  Complex. 

b.  Knowledge  Area  Expertise  Availability  can 
be  Favorable,  Neutral,  or  Unfavorable. 

c.  Organizational  Units  Involved  can  be  any 
number. 

d.  Company  Management  Involved  can  be 
Favorable,  Neutral,  or  Unfavorable. 

e.  Organizational  Environment  Complexity  can 
be  Favorable,  Neutral,  or  Unfavorable. 

f.  Computer  Expertise  Requirements  can  be 
Favorable,  Neutral,  or  Unfavorable. 

g.  Computer  Expertise  and  Availability  can 
be  Favorable,  Neutral,  or  Unfavorable. 

h.  Computer  Expertise  Adequacy  can  be  Good, 
Okay,  or  Poor. 
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Sj _ Analysis  Phase  The  analysis  phase  requires 

decomposing  the  decision  situation  under  study  into  the 
smallest  manageable  pieces  possible.  The  information  from 
this  decomposition  process  will  be  documented  in  block 
diagrams  known  as  decision  situation  diagrams  (which  model 
the  specific  area  under  study) . 

1.  Intermediary  Knowledge  Acquisition 

The  goal  of  knowledge  acquisition  and 
representation  is  the  transfer  and 
transformation  of  problem-solving  and  decision¬ 
making  expertise  from  some  knowledge  source 
into  a  form  useful  for  developing  a  knowledge- 
based  system.  (Mockler  and  Dologite,  1992:237) 

The  next  area  to  address  is  which  strategy  for  general 
knowledge  acquisition  is  best  for  the  problem  at  hand. 

There  are  two  overall  strategies  for  knowledge  acquisition 
according  to  Mockler  and  Dologite.  These  strategies  differ 
in  the  areas  of  the  basic  nature  of  the  interaction  and  the 
timeframe  allowed  for  the  knowledge  acquisition. 

There  are  three  types  of  interactions  possible  between 
a  knowledge  engineer  and  his  knowledge  source:  interaction 
between  the  knowledge  engineer  and  a  domain  expert; 
interaction  between  the  knowledge  engineer  and  written  or 
other  knowledge  sources;  or  interaction  between  a  machine 
and  the  knowledge  sources.  Knowledge  acquisition,  for  the 
KBS  prototype  will  be  gained  through  a  series  of  interviews 
with  domain  experts  from  both  the  LCOM  and  TAG  THUNDER 
systems  within  ASC/XRE. 
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An  important  question  to  be  asked  during  these 
interviews  * are  how  the  domain  expert  uses  a  given  strategy 
to  solve  a  certain  problem.  These  interviews  will  cover  the 
two  types  of  knowledge — structured  and  unstructured. 
Structured  knowledge  is  that  which  is  gained  through  the 
formal  education  process  or  reading  books.  Unstructured 
knowledge  (also  known  as  heuristics)  is  gained  from  on-the- 
job  experiences  the  expert  has  had  or  has  been  passed  on  by 
others . 

For  the  research  at  hand,  the  unstructured  interview 

method  was  selected  based  upon  the  following: 

An  unstructured  interview  is  used  to  start  many 
knowledge  acquisition  tasks,  since  it  can  be 
effective  in  exploring  the  background  knowledge 
involved  in  a  situation.  During  an  unstructured 
interview,  a  knowledge  engineer  actively  questions 
the  expert,  for  example,  by  asking  spontaneous 
questions  as  the  expert  is  performing  a  task. 

(Mockler  and  Dologite,  1992:238) 

For  the  unstructured  interview,  the  researchers  came  up 
with  a  number  of  general  questions  covering  the  process  of 
solving  a  COEA.  The  questions  are  as  follows: 


a.  What  rules/regulations/procedures  (formal 
and  informal)  are  used  every  time  that  require 
information  to  be  answered? 

b.  What  kinds  of  data  or  knowledge  is  needed 
to  reach  the  decisions  required  by  the  COEA? 

c.  Describe  a  typical  problem  for  each 
decision? 

d.  What  are  the  critical  factors  or  conditions 
that  need  to  be  met?  (i.e.  type  of  aircraft 
used  in  COEA) . 

e.  Steps  that  occur  when  you  receive  a  COEA? 
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2 .  Decision  Situation  Diagrams 
As  the  area  under  study  is  further  analyzed  and 
evaluated/  the  original  decision  situation  diagrams  are 
refined  to  get  a  more  precise  picture  of  how  the  decision  or 
task  under  study  is  accomplished. 

C.  Transformation  Phase  This  phase  is  where  the 
decision  situations  diagrams  are  "transformed"  into 
dependency  diagrams  (which  indicate  the  interrelationships 
among  critical  factors,  input  questions,  rules,  values,  and 
recommendations  made  by  the  KBS  prototype)  and  decision 
tables  (which  are  the  final  major  step  within  the  modeling 
analysis) .  From  this  paper  model,  the  actual  code  for  the 
KBS  will  be  written. 

D .  Implementation  Phase  In  this  phase  the  developer 
translates  the  paper  models  of  the  IF-THEN  Rules  and  the 
user  interface  screens  into  a  computer-based  knowledge  base 
format.  This  translation  is  accomplished  using  an  approach 
known  as  operational  prototyping. 

According  to  Turban,  "Prototyping  refers  to  a  process 
of  building  a  'quick  and  dirty'  version  of  information 
systems"  (Turban,  1990:195).  Operational  prototyping 
combines  the  rapid  results  of  the  throwaway  prototype 
approach  with  the  stability  offered  by  the  evolutionary 
prototype  approach.  An  evolutionary  prototype  is  built 
during  the  translation,  implementing  only  the  specifications 
that  are  well  understood. 


23 


Once  this  translation  is  complete,  the  developer  runs 
trial  consultations  to  produce  a  debugged  baseline  knowledge 
base.  During  these  trial  consultations  the  prototyper  also 
validates  the  inference  rules  of  the  knowledge  base.  These 
rules  should  depict  the  correct  premises  of  the  applications 
as  they  were  described  to  the  prototyper  during  the 
interviews  with  the  domain  expert. 

E .  Testing  Next,  this  baseline  is  used  by  the 
prototyper  in  a  validation  test.  In  order  to  conduct  the 
validation  test  both  the  prototype  and  the  expert  will  be 
given  a  set  of  test  cases  to  solve.  The  solutions  from  both 
the  prototype  and  the  expert  will  be  compared.  Whenever  a 
discrepancy  is  discovered  between  the  results,  the 
discrepancy  will  be  resolved  and  the  test  case  reran. 

Summary 

The  successful  building  of  this  information  system 
will  prove  the  concept  that  data  can  indeed  be  shared  within 
the  COEA  process  in  XRE.  This  KBS  prototype  will  then  be 
the  key  for  XRE  to  go  into  a  full  scale  development  of  an 
evolutionary  prototype  information  system.  This  prototype 
will  be  the  vehicle  to  share  the  information  with  the  other 
simulation  models.  By  following  the  operational  prototyping 
approach,  the  research  prototype  information  system  will 
become  the  throwaway  used  to  validate  the  concept  of  sharing 
information . 
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TABLE  1 


STEPS  TO  KBS  DEVELOPMENT 


&tSP _ Phase _ Tangible  Products. 


Isolate  Area 
for  KBS 

Development 

Project  Planning 

Bloclc  Diagram  of 
Area  Under  Study 

Target  a 

Decision  to  be 
Prototyped 

Analysis 

Decision 

Situation 

Diagrams  with 
Critical  Factors 

Create  Decision 
Tables 

Transformation 

Decision  Tables 

Create 

Dependency 

Diagrams 

Transformation 

Dependency 

Diagrams 

Write  IF-THEN 

Rules 

Transformation 

IF-THEN  Rules 
(Paper  Model) 

Design  User 
Interface 

Transformation 

Paper  Model  of 

User  Interaction 
Screens 

Enter  Knowledge 

Base  into 
computer 

Implementation 

Computer  Based 
Knowledge  Base 

Run  Trial 
Consultation 

Implementation 

Debugged  Baseline 
Knowledge  Base 

Test  and  Validate 

Testing 

The  Prototype  KBS 

(Hayes-Roth,  1983:139)  (Waterman,  1986:137) 
(Mockler  and  Dologite,  1992:104) 


25 


In  this  chapter  the  researchers  will  present  the 
results  and  findings  from  their  use  of  the  methodology 
described  in  Chapter  3 .  Each  of  the  areas  from  Chapter  3 
are  discussed  below  . 

Results  Of  Applying  MethodQlogy 

The  results  of  each  of  the  objectives  mentioned  in 
Chapter  1  will  be  demonstrated  within  this  section  of  the 
chapter . 

Objective  1.  Identify  the  information  that  is  commonly  used 
by  the  two  models. 

The  information  that  is  common  to  both  of  the  models 
was  identified  through  a  structured  interview  with  both 
Mr.  Cronk  and  Captain  McCormick.  The  results  of  the 
interview  process  can  be  found  in  Appendix  G,  III.  COEA 
Gatherer . 

Objective  2. _ Develop  the  specific  form  the  information  will 

be  in  when  passed  to  the  two  models. 

The  specific  form  was  developed  and  prototyped  within 
each  KBS  prototype.  The  final  form  for  this  information 
will  be  determined  by  the  developers  from  ASC  when  they 
build  the  full-scale  KBS. 
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ODiectiive  J.  laentiity  tne  assumptions  maae  py  eacn  moaei  as 
to  the  meaning  of  the  information  and  how  it  will  be  used,  in 
thg  mQdgl. 

The  assumptions  for  each  common  area  are  the  various 
possible  answers  for  each  of  the  questions  used  in  the  COEA 
Gatherer.  Please  refer  to  Appendix  G,  III.  COEA  Gatherer 
for  these  possible  answers.  Each  model  uses  the  information 
gathered  to  produce  their  specific  portion  of  the  COEA. 

ObjggtiVg  4t _ Identify  the  various  integrated  scenarios  that 

ggLUld  dgrivg  thg  nsedad  jdata-t 

The  nature  of  the  COEA  process  is  to  analyze  any 
potential  acquisition  or  modification  to  a  weapon  system, 
based  upon  the  proposed  scenario.  Experts  like  Mr.  Cronk 
and  Captain  McCormick  ensure  the  proposed  scenarios  "fit" 
before  applying  their  specific  simulation  model  to  produce 
the  COEA  results.  Therefore,  any  scenario  proposed  by  the 
user/sponsor  of  the  COEA  will  "fit"  this  objective. 

tivg  5. _ Develop  a  prototype  information  system. 

In  Table  1  of  Chapter  3,  the  nine  steps  of  the  KBS 
Development  Process  Steps  are  listed.  As  Mockler  and 
Dologite  noted  in  their  book,  "the  design  methodology  can  be 
applied  to  whatever  shell  is  used"  (1992:103).  The  steps 
are  as  follows: 

A.  Isolate  the  Area  for  KBS  Development. 

Create  a  block  diagram  of  the  area  under  study. 

It  should  indicate  the  sub-area  selected  for  the  initial  KBS 
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prototype  development.  This  block  diagram  is  found  in 
Chapter  1  as  Figure  1 . 

B.  Target  a  Decision  to  be  Prototyped. 

Create  a  block  diagram  of  the  exact  decision 
situations  to  be  prototyped.  Through  the  use  of  the 
prototyping  methodology,  the  researchers  discovered  a  need 
for  a  third  area  within  the  COEA  process.  Therefore,  there 
are  three  specific  situations  to  be  prototyped.  The  three 
situations  are  LCOM,  TAC  THUNDER,  and  the  COEA  Gatherer. 
Each  block  diagraun  should  indicate  the  critical  factors 
necessary  to  make  a  recommendation.  These  block  diagrams 
are  found  as  Figures  3  through  7 . 

C.  Create  Decision  Tables. 

These  decision  tables  should  indicate  all  input 
questions,  rules,  values,  and  recommendations  made  by  the 


Figure  3.  Decision  Situation  Diagram  (Level  1) 
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KBS  prototypes.  The  decision  tables  for  LCOM,  TAG  THUNDER, 
and  COEA  Gatherer  are  found  in  Appendix  C. 
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Figure  6.  Decision  Situation  Diagram  —  TAG  THUNDER 


Figure  7.  Decision  Situation  Diagram  —  COEA  Gatherer 


D,  Create  a  Dependency  Diagram. 

Convert  the  final  block  diagram  from  Step  B  into  a 
dependency  diagram.  Each  triangle  represents  a  decision 
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table  from  the  earlier  steps.  These  dependency  diagrams  are 
found  as  Appendix  D. 

E.  Write  IF-THEN  Rules. 

Convert  the  reduced  decision  tables  to  IF-THEN 
rules .  The  IF-THEN  rules  for  the  KBS  are  found  in 
Appendix  E. 

F.  Construct  the  User  Interface. 

These  segments  are  the  parts  a  user  sees  when 
running  a  consultation.  These  screens  usually  consists  of 
the  opening  and  closing  messages  for  the  consulting  session. 
EXSYS  Professional  does  not  currently  have  the  capability  of 
the  "Print  Screen"  to  allow  the  printing  of  each  individual 
consultation  screen.  The  user  interface  messages  (Opening 
and  Closing)  however  can  be  found  in  Appendix  F. 

G.  Enter  the  Knowledge  Base  into  the  Computer. 

Using  the  expert  system  shell  "editor",  type  the 

elements  that  constitute  the  "knowledge  base"  into  the 
computer  file.  This  knowledge  base  is  found  in  Appendix  G. 

H.  Run  a  Trial  Consultation. 

If  errors  prevent  a  smooth  run,  the  developer  will 
"debug"  the  errors  using  the  editor.  This  "debugging" 
process  often  takes  several  iterations  to  rid  the  file  of 
all  latent  "bugs".  With  EXSYS  Professional,  rather  than 
manually  debugging  the  KBS,  the  developer  can  validate  the 
KBS.  Th**3e  validations  can  be  accomplished  by  using  either 
systematic  or  random  testing  method. 
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Systematic  testing  allows  all  possible  combinations 
of  input  to  be  tested  for  a  variety  of  possible 
errors.  If  the  expert  system  is  large,  and 
systematic  testing  the  entire  system  would  take  too 
long,  systematic  testing  of  portions  of  the  system 
or  random  testing  of  the  entire  system  can  be 
performed.  (EXSYS  Manual:  C102) 


There  are  three  different  validation  files  that  are 
created  every time  the  KBS  is  run  through  the  validation 
option.  The  first  is  the  branched  tree  diagram,  which 
conveniently  displays  the  overall  KBS  structure  for  each 
choice  selected. 

The  second  type  of  tree  diagram  is  the  linear,  which 

displays  all  the  values  from  each  node  for  a  specific  branch 

displayed  in  the  first  validation  diagram.  The  third 

validation  is  the  error  file,  which  contains  any  reports  of 

detected  errors  and  the  input  that  produced  the  error. 

The  validation  function  will  detect  and  report 
combinations  of  input  that: 

1.  Produced  no  conclusions 

2.  Failed  to  derive  needed  qualifiers  or  variables 
that  should  be  derived 

3 .  Created  error  loops 

4.  Assigned  a  variable  a  value  which  is  outside  of 
the  limits  specified  for  the  variable 

5.  Assigned  more  values  to  a  qualifier  than  the 
maximum  number  allowed  for  that  qualifier 

6.  Special  custom  tests  designed  with  the  report 
generator.  (EXSYS  Manual:  C104) 
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I.  Test  and  Validate  the  Prototype. 

Test  case  scenarios  that  were  used  to  test  and 
validate  the  prototypes  were  an  on-going  real-world  COEA 
project.  Using  the  KC-135  Multi-Point  Refueling  COEA  as  the 
test  case,  the  sponsors  responded  to  questions  posed  by  the 
COEA  Gatherer  KBS  prototype.  The  results  were  automatically 
saved  to  a  file.  The  final  results,  which  match  exactly  the 
conclusions  drawn  by  the  experts,  are  available  in 
Appendix  I.  Both  Mr.  Cronk  and  Captain  McCormick  agreed 
that  these  KBSs  will  be  valuable  tools  when  fully  developed. 

Preliminary  Screening  In  the  area  of  Preliminary 
Screening,  Mockler  and  Dologite  (1992:47-48)  stated  several 
questions  need  to  be  answered  as  part  of  the  preliminary 
screening  process. 

1.  Do  recognized  experts  exist?  The  Department  of 
the  Air  Force  has  entrusted  both  Mr.  Cronk  and  Captain 
McCormick  with  the  management  and  leadership  of  each  of  the 
two  simulation  models,  LCOM  and  TAC  THUNDER.  If  anyone 
within  ASC  has  any  questions  about  these  systems,  these  are 
the  people  they  turn  to  for  the  answers. 

2_.  ^an  the  experts  do  the  task  better  than 
amateurs  and  can  their  skills  be  taught  to  others?  The 
experts  have  developed  the  answers  for  each  COEA  and  need  to 
have  this  information  codified  and  replicated  within  a 
knowledge  based  system  (KBS)  to  further  integrate  their 
ability  to  analyze  and  answer  questions  much  more 
efficiently. 
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X. _ Do  different  experts  agree  on  solutions?  An 

Important  element  of  COEAs  is  that  since  they  "often  involve 
very  different  systems  advocated  by  different  services  or 
commands,  this  process  provides  a  disciplined  approach  for 
comparing  concepts"  (AFSCP  173-1:1).  This  process  makes  the 
different  experts  "agree"  as  well  as  experts  with  differing 
opinions  are  able  to  reach  a  consensus  of  opinion. 

_ Does  the  task  require  reason  and  informed 

iudqments.  as  opposed  to  mere  common  sense?  Yes,  the  task 
at  hand  requires  "information  and  supporting  documentation 
from  the  COEA  process  (that)  is  critical  for  selecting  the 
best  possible  system  to  satisfy  user  requirements"  (AFSCP 
173-1:2).  Such  a  requirement  cannot  be  derived  from  common 
sensical  approaches  to  any  COEA. 

5.  Is  the  task  well  understood?  Besides  the  AFSC 
pamphlet  for  guidance,  COEAs  have  been  accomplished  since 
late  1989.  Both  Mr.  Cronk  and  Captain  McCormick  can  provide 
additional  information  about  all  aspects  of  the  COEA  from 
their  functional  area.  As  ASC/XREC  members,  both  have  been 
around  the  COEA  process  since  its  inception. 

L, — Can  the  experts  articulate  their  methods?  The 
experts  have  been  forced  to  articulate  their  methods  through 
such  avenues  as  DoD  Directives  5000.1  and  .2,  the  AFSC 
pamphlet,  and  the  former  Strategic  Air  Command's  COEA 
implementation  plan.  There  have  been  more  data  gained 
through  several  interviews  and  suggested  readings  on  LCOM, 
TAC  THUNDER,  and  COEAs. 
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7.  I3  the  task  of  manageable  size?  The  task  will 

* 

be  scoped  to  a  manageable  size  to  allow  the  prototype  to  be 
built . 

8.  Are  typical  example  cases  or  situations 
readily  available?  The  COEA  process  has  been  well  defined 
and,  as  previously  mentioned,  there  are  6  COEAs  currently 
ongoing  at  this  point  in  time. 

In  area  A  4,  the  Decision  Situation  Diagram  for 
Estimating  KBS  Project  Proposal  Level  of  Risk  (Initial 
Prototype  Phase) ,  the  following  answers  were  developed 
through  the  knowledge  acquisition  and  interview  processes 
used  by  the  researchers. 

a.  Knowledge  Area  Complexity  is  Complex 

b.  Knowledge  Area  Expertise  Availability  is 
Favorable 

c.  Organizational  Units  Involved  *  2 

d.  Company  Management  Involved  =  Favorable 

e.  Organizational  Environment  Complexity  is 
Favorable 

f.  Computer  Expertise  Requirements  is  Favorable 

g.  Computer  Expertise  and  Availability  is 
Favorable 

h.  Computer  Expertise  Adequacy  is  Good 

The  determined  level  of  risk  for  the  development  of  the 
initial  prototype  phase  is  Favorable. 
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Limitations  of  the  Results 

During  the  process  of  applying  the  prototyping 
methodology,  the  researchers  discovered  the  problem  at  the 
foundation  of  the  COEA  process.  Although  the  KBSs  developed 
will  improve  the  information  sharing  within  the  COEA 
process,  a  redefining  of  the  overall  process  is  essential 
before  all  the  potential  benefits  can  be  realized.  The 
original  problem  scope  stated  that  it  was  a  lack  of 
information  sharing  within  the  COEA  process . 

During  the  unstructured  interviews  with  both  experts 
present  it  was  discovered  that  the  process  is  not  well 
defined.  This  lack  of  proper  process  definition  has  led  to 
disconnects  in  the  effectiveness  and  efficiency  of  the 
dissemination  of  information  necessary  to  the  COEA  process. 
This  discovery  is  outside  the  scope  of  the  initial  research 
effort. 

Summary 

An  overall  review  of  the  results  shows  that  the  use  of 
the  KBSs  as  the  solution  to  the  problem  of  information 
sharing  is  appropriate.  When  the  process  has  been  redefined 
and  the  KBSs  fully  implemented  within  the  process,  the 
benefits  of  the  solution  will  be  fully  realized. 
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V.  Conclusions  and  Reconunendations 


Introduction 

This  chapter  draws  conclusions  from  the  results 
presented  in  Chapter  IV.  Based  upon  these  results, 
recommendations  are  made  and  then  areas  for  further  research 
are  described. 

Conclusions 

Discovering  the  lack  of  process  definition  within  the 
COEA  process  while  understanding  the  potential  for  the  COEA 
process  indicates  the  critical  need  for  process  improvement. 
The  biggest  flaw  within  the  current  process  flow  is  the  lack 
of  coordination  which  is  manifested  in  the  continual 
meetings  and  conferences  to  resolve  conflicts  within  the 
process  areas .  The  areas  affected  by  this  lack  are  the 
initial  scoping  of  the  COEA  solution  process,  the  model- 
specific  data  collection,  and  the  generation  and  integration 
of  the  model  results  into  the  final  COEA  analysis  report. 
This  same  lack  of  coordination  makes  the  process  inefficient 
by  not  properly  using  all  the  assets  available  to  the 
process . 

In  the  manpower  realm  alone  there  are  demonstrable 
inefficiencies  within  the  current  COEA  process.  People  are 
misused  in  two  ways .  First,  the  people  requested  to  attend 
the  continual  meetings,  and  conferences  may  receive  only  a 
few  minutes  worth  of  pertinent  information,  thereby  wasting 
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the  rest  of  their  time  at  the  meeting.  Second,  people  are 
not  requested  to  attend  meetings  when  they  should  be 
included.  This  means  these  individuals  miss  a  chance  to 
receive  the  needed  information  first-hand  and  then  to  avail 
themselves  of  the  opportunities  to  request  other  needed 
information  at  the  appropriate  moment. 

Another  realm  where  inefficiency  currently  exists  is 
the  waste  of  available  information.  Since  no  formal  or 
informal  "information  networks"  exist  within  the  COEA 
process,  needed  information  and  cross-functional  expertise 
are  not  used.  Only  after  the  experts  from  LOOM  and  TAG 
THUNDER  met  for  the  discussion  and  unstructured  interviews 
for  the  KBSs,  did  the  realization  of  the  potential  benefits 
of  networking  their  common  information  become  evident.  This 
artificial  communications  barrier  is  a  holdover  from  the 
days  before  XR  was  integrated  to  better  support  the 
Integrated  Product  Development  concept.  Business 
reorganization  alone  is  never  enough  to  improve  the 
processes  found  within  an  organization. 

Recommendations 

The  first  recommendation  is  to  redefine  the  process 
using  a  functional  process  improvement  methodology.  In 
1992,  the  Director  of  Defense  Information  for  OASD  issued 
the  Interim  Management  Guidance  on  Functional  Process 
Improvement  (DoD  8020. 1-M).  This  guidance  provides  "DoD 
functional  managers  with  the  processes  and  procedures  that 
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should  be  applied  when  conducting  process  improvement 
projects  throughout  the  DoD"  (CIM,  1993 ;v) .  Without  this 
process  improvement,  no  tool  or  organizational  framework  can 
be  successfully  implemented. 

Only  after  the  process  has  been  redefined  and 
restructured  should  any  tools  be  selected  or  built  to 
support  the  process.  Once  the  process  has  been  redefined 
and  well  understood,  the  environment  will  be  able  to  reap 
all  the  potential  benefits  of  the  researcher' s  proposed 
solution.  The  process  redefinition  will  help  in  fully 
defining  those  specific  areas  where  a  KBS  can  be  utilized  to 
enhance  the  information  gathering  process  necessary  for  the 
COEA. 

Further  Research 

Areas  for  further  research  from  the  concept  of 
information  sharing  are  enormous.  Theses  could  be  developed 
from  the  following:  developing  the  full-scale  versions  of 
all  three  of  the  KBSs  prototyped  within  this  study,  using  a 
functional  improvement  methodology  to  redefine  and 
restructure  the  COEA  process  as  found  here  at  WPAFB,  and 
integrating  and/or  standardizing  the  COEA  processes  between 
each  separate  Air  Force  activity  that  uses  the  COEA  process . 

All  of  these  suggested  theses  have  the  potential  to 
help  shape  tomorrow's  Air  Force  in  a  significant  way.  Each 
thesis  idea  could  certainly  be  a  sponsored  effort,  ensuring 
the  proper  level  of  support  necessary  to  produce  a  quality 
thesis . 
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.  A  computer  program  that  provides  features  and 


functions  particular  to  the  user's  information  needs 
(Kroenke,  1992:777). 

Artificial  Intelligence.  The  capability  of  a  device,  such 
as  a  computer,  to  perform  functions  or  tasks  that  would  be 
regarded  as  intelligent,  if  they  were  observed  in  humans 
(Mockler  and  Dologite,  1992:772). 

Domain  Expert.  An  individual  who  is  highly  recognized  as 
having  the  knowledge  and  know-how  necessary  to  solve  a 
problem  or  make  a  decision  in  a  specific  knowledge  domain 
(Mockler  and  Dologite,  1992:773). 

Effectiveness .  The  attainment  of  a  predetermined  goal.  The 
degree  to  which  a  predetermined  goal  is  met  (Horngren  and 
Foster,  1991:943). 

Efficiency.  The  relationship  between  the  inputs  used  and 
outputs  achieved.  The  fewer  the  inputs  used  to  attain  a 
given  output,  the  greater  the  efficiency  (Horngren  and 
Foster,  1991:943). 

Ergonomics  (Also  known  as  Human  Factors).  Discovers  and 
applies  information  about  human  behavior,  abilities, 
limitations,  and  other  characteristics  to  the  design  of 
tools,  machines,  systems,  tasks,  jobs,  and  environments  for 
productive,  safe,  comfortable,  and  effective  human  use 
(Sanders  and  McCormick,  1993:5). 


Evolutionary  Prototyping.  A  specific  type  of  prototyping 
that  is  used  to  uncover  unknown  requirements  and  continues 
to  evolve  into  the  fully  functional  system  (Davis,  1992:71), 
Expert  System.  A  general  term  used  to  refer  to  knowledge- 
based  systems,  and  to  describe  a  wide  range  of  advanced 
computer  systems  variously  described  as  decision  support 
systems,  executive  information  systems,  management 
information  systems,  and  executive  support  systems  (Mockler 
and  Dologite,  1992:774) . 

Heuristics .  Rules  of  thvimb  or  other  strategies  used  in 
problem-solving  or  decision  making  (Mockler  and  Dologite, 
1992:774) . 

Information  System.  An  open  purposive  system  that  produces 
information  using  the  input /process /output  cycle.  The 
minimal  information  system  consists  of  people,  procedures, 
and  data  (Kroenke,  1992:782). 

Interoperability .  The  capacity  to  integrate  technology 
between  or  among  different  technical  platforms  (CIM, 
1993:159) . 

Knowledge-Based  System.  A  computer  system  that  attempts  to 
replicate  specific  human  expert  intelligent  activities 
(Mockler  and  Dologite,  1992:774). 

Knowledge  Domain.  A  field  of  knowledge  that  can  be  defined 
by  scope,  range,  depth,  and/or  breadth  (ibid) . 

Knowledge  Engineer.  An  individual  who  accomplishes  KBS 
development  jobs  of  situational  analysis  and  representation, 
and  computer  system  design  and  implementation  (ibid) . 
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Operational  Prototyping.  A  prototyping  approach  that 
combines  the  rapid  results  of  the  throwaway  prototype  and 
the  stability  of  the  evolutionary  prototype  (Davis, 

1992:73) . 

Throwaway  Prototyping.  A  prototyping  approach  that  is  used 
to  discover  which  requirements  are  real  and  which  are  not. 
This  prototype  is  discarded  after  the  desired  information  is 
learned  (ibid: 71) . 
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Appendix  B:  Answers  from  the  Unstructured  Interviews 
I .  LCOM 

The  researchers  asked  all  of  the  questions  to  the 
expert  so  as  to  give  him  an  idea  of  the  direction  the 
interview  would  take.  The  expert  stated  that  the  steps  that 
occur  in  whenever  a  COEIA  is  received  would  be  the  best 
framework  to  answer  all  the  other  questions, 

5 .  Steps  that  occur  when  you  receive  a  COEA? 

First,  all  the  representatives  are  called  together  in  a 
big  meeting  to  decide  whether  XRE  can  produce  the  required 
answers  for  the  COEA  Request.  Some  of  the  initial  screening 
questions  are: 

—  Does  XRE  have  a  valid  scenario  to  fit  this  COEA? 

—  Does  XRE  have  applicable  databases  to  fit  this 
scenario  (i.e.  a  Campaign /Threat  model  for  TAC  THUNDER,  a 
Supportability  model  for  LCOM,  a  Cost  model,  etc...)? 

4 .  What  are  the  critical  factors  or  conditions  that 
need  to  be  met  (i.e.  type  of  aircraft  used  in  COEA)? 

The  overall  area  for  critical  factors  would  be  in 
understanding  the  proposed  modification.  Each  main  area  of 
concern  would  be  a  Description  of  the  Modification,  LCOM 
database  for  the  specific  aircraft  that  the  modification  is 
proposed  for,  a  database  for  the  scenario  to  be  modeled,  and 
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familiarization  and  debugging  of  the  database  within  the 
LCOM  simulation  model  framework. 

Description  of  Modification: 

-  Does  the  COEA  Request  have  a  Engineering  Change 
Proposal  (ECP) ? 

—  Requires  a  Yes  and  No  branching.  The  Yes 
branch  should  then  flow  into  next  D  of  M  question.  The  No 
branch  should  specify  that  the  validity  of  the  COEA  needs  to 
be  verified  with  the  System  Program  Office  (or  other 
sources)  to  get  a  copy  of  the  ECP. 

-  Is  the  Logistics  Support  Analysis  data 
available? 

—  Each  question  within  this  section 
requires  a  Yes  and  No  branching.  The  Yes  branch  should  skip 
the  subsequent  possible  sources  of  data  for  the  model  and 
proceed  to  the  next  area  Task  Times  for  Maintenance.  The  No 
branch  should  then  flow  into  the  next  possible  source  for 
model  data. 

-  Existing  data  on  comparable  equipment? 

-  Existing  data  in  historical  database? 

-  Existing  data  in  other  databases  (i.e.  Navy, 
commercial,  etc)? 

Task  Times  for  Maintenance: 

-  Sequence  of  maintenance  tasks? 

—  Each  question  within  this  section  requires 
a  Yes  and  No  branching.  The  Yes  branch  should  flow  into  the 
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next  question  within  this  section  since  all  these  questions 
are  required  for  the  LOOM  model.  The  No  branch  should  also 
flow  into  each  question  since  this  checklist  is  to  determine 
which  information  is  at  hand  and  which  is  needed. 

-  Resource  requirements? 

-  Task  times  for  completion  of  task? 

-  Maintenance  crew  size  data? 

-  Support  facilities? 

-  Reliability  values? 

LCOM  Database  (DB)  Familiarization/Debug: 

-  Is  there  an  LCOM  DB  with  specific  aircraft  for 
the  scenario? 

—  Each  question  within  this  section  requires 
a  Yes  and  No  branching.  The  Yes  branch  then  flows  into  the 
next  question  within  this  section.  The  No  branch  then  asks 
if  there  is  a  DB  available  from  other  sources. 

-  Is  DB  available  in-house? 

-  Other  sources  of  DB  (MAJCOM,  Navy,  commercial, 
generic) ? 

When  DB  is  Available: 

-  Compare  to  current  COEA  scenario  by  checking  the 
same  flying  schedule  (peace/wartime/other) ? 

—  Each  question  in  this  section  requires  a 
Yes  and  No  branching.  The  Yes  branch  will  flow  into  the 
next  question  from  within  this  section.  The  No  branch  will 
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also  flow  into  the  next  question  to  determine  which  portions 
of  the  DB  need  to  be  modified  to  fit  the  current  scenario. 

-  Same  types  of  missions? 

DB  Integrity  Check: 

-  If  DB  has  never  been  run  through  LOOM  simulation 
model.  Then  run  DB  through  the  simulation  model.  This  is 
the  quickest  way  to  check  integrity  of  the  DB. 

—  If  integrity  of  the  DB  is  verified,  then 
the  information  gathering  process  is  complete. 

—  If  integrity  is  bad,  determine  from  the 
error  messages  from  LCOM  simulation  model  what  the  magnitude 
of  the  data  errors  are.  If  possible,  fix  the  errors.  If 
not,  then  DB  will  be  rejected  and  another  sought  out  or 
built. 

The  following  questions  were  never  readdressed  by  the 
researchers  since  they  all  were  answered  by  the  previous 
answers  and  scenarios  as  presented  by  the  expert. 

1.  What  rules/regulations/procedures  (formal  and 
informal)  are  used  everytime  that  require  information  to  be 
answered? 

2.  What  kinds  of  data  or  knowledge  is  needed  to  reach 
the  decisions  required  by  the  COEA? 

3.  Describe  a  typical  problem  for  each  decision? 
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Unstructured  Interview 
Captain  Dave  McCormick  9  Jul  93 
Steps  in  the  COEA  Process . . . 

Critical  Factors 

A.  Establish  Year/Timeframe 

Is  the  scenario  consistent/nonconsistent? 

Have  several  years  over  the  lifetime  of  the  system 
been  indicated? 


B.  Select  proper  Defense  Planning  Guidance  (DPG)-based 
scenario  for  COEA  effectiveness  analysis 

Are  the  scenarios  traceable  back  to  DPG/IPS? 

Do  the  scenarios  seem  contrived? 

Do  the  scenarios  identify  the  mission  tasking  for 
the  alternatives? 

Does  one  of  the  scenarios  provide  a  stressful 

case? 


Does  one  of  the  scenarios  provide  an  unlikely 

case? 

Does  one  of  the  scenarios  provide  a  likely  case? 
Do  the  scenarios  present  a  good  operational  range 
of  possibilities? 
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C.  Establish  a  baseline  DB  for  TT  (at  least  2  theaters 
should  be  examined) 

If  only  one  theater  selected,  has  the  rationale 
been  documented? 

Are  the  DPG/IPS  scenarios  built  on  the  validated 

threat? 

Are  blue  systems  a/c  &  ground  assets  correctly 
modeled  (performance,  lethality,  sortie  rates,  etc)? 

Has  user/sponsor  reviewed  proposed  scenario  DB? 
Are  changes  required? 

If  yes,  incorporate  changes. 

D.  Define  Functional  Objectives  (FOs) 

Have  the  mission  tasks  been  identified  for  the 
system  based  on  the  need? 

(Ensure  model  generates  appropriate  data 
based  on  mission  need) 

Are  the  mission  tasks  quantifiable? 

(Has  the  mission  task  been  quantified?) 

E.  Concept  of  operations  (CONOPs) 

Is  the  employment  of  the  system  feasible? 

Is  the  operations  and  maintenance  force  structure 

valid? 

Are  interfaces  with  other  systems  considered? 
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F.  List  of  Alternatives 

Is  each  alternative  described  in  detail? 

Has  the  (current  system)  baseline  case  been 
identified? 

Has  an  adequate  range  of  alternatives  been 
identified? 

Do  the  alternatives  consider  changes  in 
requirements? 

Are  the  current  or  prospective  systems  reasonable 
alternatives? 

Have  you  explained  the  rationale  for  non-selection 


of  alternatives? 


Appendix  C:  Decision  Tables 


I.  LOOM  Prototype 


Rule 
A  1 

A  2 

A  3 

A  4 

A  5 

A  6 
A  7 

A  8 

A  9 
A  10 
A  11 

A  12 


1.  Description  of  Modification 


Qualifier 

Choice 

Recommendation 

ECP  Provided  w/  COEA 

Y/N 

Proceed/ Check 
validity 
of  COEA 

LSA  A/C  data  avail? 

Y/N 

Proceed/ Check 
other  sources 

Other  AF  MNX  data  avail? 

Y/N 

Proceed/Check 
other  sources 

Historical  AF  DB  avail? 

Y/N 

Proceed/Check 
other  sources 

Other  Non-AF  DB  avail? 

Y/N 

Proceed/COEA  not 
possible 

Or  any  one  is  Y 

Y 

Proceed 

If  all  are  N 

Stop  COEA  warning 
message 

2 .  Task  Times  For  MNX 


Sequence  Mnx  Tasks  avail? 

Y/N 

Proceed/Look  for 
comparable  data 

Resource  reqs  avail? 

Y/N 

Proceed/Ask  expert 

Task  time  for  each  task? 

Y/N 

Proceed/Ask  expert 

Mnx  Crew  specs  avail? 

Y/N 

Proceed/Look  for 
comparable  data 

Support  facilities  specs? 

Y/N 

Proceed/Look  for 
comparable  data 
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A  13 

A  14 

A  15 

A  16 

A  17 

A  18 

A  19 

A  20 

A  21 


Reliability  values  avail?  Y/N  Proceed/Look  for 

comparable  data 


Else  resolve  any  NOs 


COEA  must  stop 


3.  LCOM  DB  Familiarization/Debuq 


Specific  LCOM  DB  scenario? 

Y/N 

Proceed/Look  at 
other  scenario 
sources 

Scenario  avail  in-house? 

Y/N 

Proceed/Look  at 
other  scenario 
sources 

Other  scenario  sources? 

Y/N 

Proceed/COEA  not 
possible 

4.  When  DB  Avail.  Comoare  to 

Scenario 

Same  flying  schedule? 

Y/N 

Proceed/Modify 

scenario 

Same  types  of  mission? 

Y/N 

Proceed/Modify 

scenario 

5 .  DB  Integrity  Check 

Has  DB  been  run  in  model?  Y/N  Proceed/Run  in 

model 

What  were  results  of  run?  G/B  Proceed/ Investigate 

errors  for 
magnitude  of  fix 
needed 
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II.  TAG  THUNDER  Prototype 

1.  Establish  Year/Timeframe 


Ri’le 

Qualifier 

Choice 

Reconunendation 

A 

1 

Scenario  year  consistent 

Y/N 

Proceed/Review 

A 

2 

Several  years  identified 

Y/N 

Proceed/Ask  expert 

2-  Select  Prooer  (DPG) -Baaed 

Scenarios  Analvsis 

A 

3 

Scenarios  traceable 

Y/N 

Proceed/Provide 

rationale 

A 

4 

Scenarios  appropriate 

Y/N 

Proceed/Review 

A 

5 

Scenarios  identify  mission 

Y/N 

Proceed/Get  listing 

A 

6 

Scenarios  stressful 

Y/N 

Proceed/Review 

A 

7 

Scenarios  likely 

Y/N 

Proceed/Review 

A 

8 

Scenarios  good  range 

Y/N 

Proceed/Consider 

other  scenarios 

_ Establish  A  Baseline  DE 

\  For  TT 

A 

9 

Theaters  more  than  one 

Y/N 

Proceed/Get  expert 

A 

10 

Scenarios  threat 

Y/N 

Proceed/Obtain  STAR 

A 

11 

Blue  assets  correct 

Y/N 

Proceed/Review 

A 

12 

Expert  reviewed 

Y/N 

Proceed/Get  expert 

4.  Define  Functional  Obiectives  (FOs) 

A 

13 

Tasks  identified 

Y/N 

Proceed/Coordinate 

A 

14 

Tasks  quantified 

Y/N 

Proceed/Develop 

5.  Conceot  Of  Ooerations 

(CONOPs) 

A 

15 

Employment  feasible 

Y/N 

Proceed/Review 

A 

16 

Structure  valid 

Y/N 

Proceed/Review 

A 

17 

Interfaces  considered 

Y/N 

Proceed/Obtain  data 
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6.  List  Of  Alternatives 


A 

18 

Alternative  in  detail 

Y/N 

Proceed/Obtain 

detail 

A 

19 

Baseline  identified 

Y/N 

Proceed/Review 

A 

20 

Alternatives  identified 

Y/N 

Proceed/Review 

A 

21 

Alts  consider  change 

Y/N 

Proceed/Review 

A 

22 

Are  Systems  alternatives 

Y/N 

Proceed/Review 

A 

23 

Non-Selection  explained 

Y/N 

Proceed/Document 
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III.  COEA  Gatherer 


Rule  Qualifier  Choice  Recommendation 


1 .  General  Requirements 


1 

Scenario  is  Peacetime 

Y/N 

Send  info  on 

2 

Scenario  is  Wartime 

Y/N 

Send  info  on 

3 

Scenario  is  other 

Y/N 

Resolve  with  user 

4 

Scenario/IOC  yr  consistent 

Y/N 

Send  info  on/ 
Review  with  user 

5 

ECP  available 

Y/N 

Send  info  on/ 
Check  validity 

30 

General  Rqmts  all  avail 

Y/N 

2.  Facilities  and  Deolovment 

6 

Num  locations  and  #  ac 
at  each  site  determined 

Y/N 

Send  info  on/ 

Get  info 

7 

Supply  concept  determined 

Y/N 

Send  info  on/ 

Get  info 

8 

Resupply  time  determined 

Y/N 

Send  info  on/ 

Get  info 

9 

Extent  of  maintenance 
capability  determined 

Y/N 

Send  info  on/ 

Get  info 

10 

Shelter  determined  at  each  site 

Y/N 

Send  info  on/ 

Get  info 

11 

Facilities  &  Support  equipment 
determined 

Y/N 

Send  info  on/ 

Get  info 

31 

All  Facilities  &  Deployment 
info  available 

Y/N 

3.  Mission  Recruirements 

12 

Mission  types  determined 

Y/N 

Send  info  on/ 

Get  info 

13 

AC  Config  for  each  mission 
determined 

Y/N 

Send  info  on/ 

Get  info 
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14  Mission  priorities  determined 


Y/N  Send  info  on/ 
Get  info 


15 

Mission  cancellation  delay  time 
tolerances  determined 

Y/N 

Send  info 
Get  info 

on/ 

16 

Mission  tasks  for  system  based 
on  need  determined 

Y/N 

Send  info 
Get  info 

on/ 

34 

All  Mission  Rqmts  info  available 

Y/N 

4.  Ooerations  and  Schedulino 

Policy 

17 

Aircraft  sortie  rates  determined 

Y/N 

Send  info 
Get  info 

on/ 

18 

Requirements  for  complementary 
missions  determined 

Y/N 

Send  info 
Get  info 

on/ 

19 

Interfaces  with  other  systems 
have  been  considered 

Y/N 

Send  info  on/ 
Review  with  user 

20 

Interface  data  is  available 

Y/N 

Send  info 
Get  info 

on/ 

21 

Interfaces  w/  other  systems 
been  considered  and  appropriate 
interface  data  is :  not  available 

Y/N 

Get  info 

22 

Nvimber  of  ac  on  alert  at  each 
site  available 

Y/N 

Send  info 
Get  info 

on/ 

35 

All  Ops  and  Sched  info  avail 

Y/N 

5.  Ground  Alert 

23 

Missions  to  be  flown  from  alert 
determined 

Y/N 

Send  info 
Get  info 

on/ 

24 

Frequency  of  alert  missions 
determined 

Y/N 

Send  info 
Get  info 

on/ 

25 

Alert  replacement  policy 
determined 

Y/N 

Send  info 
Get  info 

on/ 

32 

All  Ground  Alert  info  available 

Y/N 

6.  Maintenance  Concepts 

&  Operations 

26 

Maintenance  concept  determined 

Y/N 

Send  info 

on/ 

Get  info 
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27 

Organizational  structure  & 
maintenance  concept  match 

Y/N 

Send  org  info  on/ 
Resolve  with  user 

28 

AFSC  structure  &  org  structure 
in  compliance 

Y/N 

Send  AFSC  info  on/ 
Resolve  with  user 

29 

AFSC  structure  &  maint  concept 
level  in  compliance 

Y/N 

Send  AFSC  info  on/ 
Resolve  with  user 

33 

All  Maint  Concept  &  Ops  info 
available 

Y/N 

7 .  Overall  Recommendation 


36  General  Requirements  and 

Facilities  &  Deployment  and 
Missions  Rqmts  and 
Ops  &  Sched  Policy  and 
Ground  Alert  and 
Maint  Concepts  &  Organization 
info  avail 


Y/N  Proceed  to  send 
info  to  LCOM  and 
TAG  THUNDER/ 

Not  all  info  is 
available  for 
LCOM  and  TAG 
THUNDER  to  process 
this  COEA 
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I .  LCOM 
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FIgura  8.  0«pantf*ncy  Oiagram  l  —  LCOM 
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Igure  9.  Dependency  Dlagrem 
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61 
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Avallabltt  Information: 


The  scenario/IOC  year  is: 

The  Engineering  Change  Proposal  information  is: 

The  n\]inbers  and  locations  of  aircraft  at  each  site  are: 
The  supply  concept  is: 

The  resupply  time  is: 

The  maintenance  capability  required  at  each  site  is: 

The  shelter  information  at  each  site  is: 

Facilities  and  support  equipment  rqmts  for  each  site  are: 
The  mission  types  are: 

The  aircraft  configuration  for  each  mission  is: 

The  mission  priorities  are: 

The  tolerances  for  mission  delay  time  are; 

The  mission  tasks  for  the  system  are: 

The  aircraft  sortie  rates  are: 

The  requirements  for  complementary  missions  are: 

The  interfaces  with  other  systems  and  their  data  are: 

The  nximber  of  aircraft  on  alert  at  each  location  is: 
Missions  to  be  flown  from  alert  are: 

The  frequency  of  alert  missions  is: 

The  alert  replacement  policy  is: 

The  maintenance  concept  to  be  used  is: 

The  organization  structure  is: 

The  AFSC  structure  is: 

Information  is  not  available  or  incompatible: 

LOOM  and  TAG  THUNDER  are  not  designed  for  scenarios  other 
than  Peacetime  or  Wartime 
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See  COEA  focal  point  for  further  guidance 

Review  scenario/IOC  year  with  user  to  resolve  and/or  verify 
discrepancy 

Check  validity  of  Engineering  Change  Proposal  with  user 
Get  numbers  and  locations  of  ac  at  each  site  from  users 
Get  the  supply  concept  information  from  user 
Get  the  resupply  time  information  from  user 

Get  extent  of  mnx  capability  required  at  each  site  from  user 
Get  the  shelter  information  for  each  site  from  user 
Get  facil  &  support  eqpmnt  rqmnts  for  each  site  from  user 
Get  the  mission  types  from  user 

Get  the  aircraft  configuration  for  each  mission  from  user 

Get  the  user  to  establish  mission  priorities 

Get  user  to  determine  mission  delay  time  tolerances 

Coordinate  with  user  to  formulate  tasks  and/or  study 
measures  for  alternatives  being  examined 

Get  aircraft  sortie  rates  from  user 

Get  user  to  determine  rqmts  for  complementary  missions 
Review  the  lack  of  interface  data  with  user 
Get  data  for  interface  systems  from  user 

Get  number  of  aircraft  on  alert  at  each  location  from  user 

Get  missions  to  be  flown  from  alert  information  from  user 

Get  frequency  of  alert  missions  information  from  user 

Get  alert  replacement  policy  information  from  user 

Get  user  to  determine  maintenance  concept  to  be  used 

Get  with  user  to  rectify  organization  structure  and 
maintenance  concept  level 

Get  with  user  to  rectify  differences  in  AFSC  structure, 
organization  structure,  and  maintenance  concept  level 
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I ■  LOOM 


Appendix  E:  IF-THEN  Rules 


RULES: 


RULE  NUMBER:  1 
IF: 

the  Engineering  Change  Proposal  provided  with  the 
COEA  is  available 

THEN: 

Proceed  with  the  COEA  Infomation  Gathering  Process 
ECP  availability  is  at  --  Confidence*10/10 


RULE  NUMBER:  2 
IF: 

the  Engineering  Change  Proposal  provided  with  the 
COEA  is  not  available 

THEN: 

Check  the  validity  of  the  Proposed  COEA  with  the 
Systems  Program  Office  and  ,  if  necessary,  consult 
with  the  COEA  expert.  ECP  nonavailability  is  at  - 
Confidence«10/10 


RULE  NUMBER:  3 
IF: 

the  Logistics  Support  Analysis  aircraft  data  is 
available 

or  other  Air  Force  maintenance  data  is  available 
or  a  historical  Air  Force  Maintenance  Database  (DB)  is 
available 

or  other  Non-AF  DB  are  available 
THEN: 

Proceed  with  the  COEA  Information  Gathering  Process 
The  necessary  Analysis  Data  availability  is  at  - 
Confidence-1 0/10 


RULE  NUMBER:  4 
IF: 

the  Logistics  Support  Analysis  aircraft  data  is  not 
available 

and  other  Air  Force  maintenance  data  is  not  available 
and  a  historical  Air  Force  Maintenance  Database  (DB)  is 
not  available 

and  other  Non-AF  DB  are  not  available 
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THEN 


Check  the  validity  of  the  COEA  Proposal  with  the 
Systems  Program  Office  and  the  COEA  expert  since 
there  is  no  Analysis  Data  available  at  - 
Conf  idence=“l  0/10 


RULE  NUMBER:  5 
IF: 

the  sequence  of  maintenance  tasks  to  be  performed  is 
available 

and  the  resource  requirements  for  each  specific 
maintenance  task  are  available 
and  the  task  time  for  each  maintenance  task  to  be 
performed  is  available 

and  the  specifications  for  each  maintenance  crew  required 
' to  perform  each  maintenance  task  are  available 
and  the  support  facilities  required  for  each  maintenance 
task  to  be  performed  are  available 
and  the  reliability  values  required  for  each  maintenance 
task  to  be  performed  are  available 

THEN: 

Continue  the  COEA  Information  Gathering  Process.  The 
Specific  Task  Data  availability  is  at  -  Conf.  =  10/10 

RULE  NUMBER:  6 
IF: 

the  seq[uence  of  maintenance  tasks  to  be  performed  is 
not  available 

or  the  resource  requirements  for  each  specific 
maintenance  task  are  not  available 
or  the  task  time  for  each  maintenance  task  to  be 
performed  is  not  available 

or  the  specifications  for  each  maintenance  crew  required 
to  perform  each  maintenance  task  are  not  available 
or  the  support  facilities  required  for  each  maintenance 
task  to  be  performed  are  not  available 
or  the  reliability  values  required  for  each  maintenance 
task  to  be  performed  are  not  available 

THEN: 

Look  for  comparable  data  that  matches  the  Specified 
Task  Data  or  ask  the  COEA  expert  for  guidance  in 
finding  data  for  that  specific  task  area.  The 
nonavailability  of  the  Specific  Task  Data  is  at  - 
Conf.  »  10/10 
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<6 


RULE  NUMBER:  7 
IF: 

the  specific  LOOM  DB  scenario  is  available  in-house 

THEN: 

Proceed  with  the  COEA  Information  Gathering  Process. 
The  Scenario  availability  is  at  -  Conf.  =  10/10 


RULE  NUMBER:  8 
IF: 

the  specific  LCOM  OB  scenario  is  not  available 
in-house 

THEN: 

Check  other  sources  with  the  same  type  of  aircraft  as 
indicated  in  the  COEIA  Proposal  (like  MAJCOMs,  other 
services,  or  commercial  aviation  services)  to  get  the 
needed  Scenario  specific  data.  The  need  for  Scenario 
specific  data  is  at  -  Confidence-lO/lO 


RULE  NUMBER:  9 
IF: 

the  flying  schedule  within  the  DB  is  the  same 

THEN: 

Continue  with  the  COEA  Information  Gathering  Process. 
The  availability  of  the  DB  is  at-  Conf.  -  10/10 


RULE  NUMBER:  10 
IF: 

the  mission  types  within  the  scenario  to  be  used  are 
the  same 

THEN: 

Continue  with  the  COEA  Information  Gathering  Process. 
The  availability  of  the  DB  is  at-  Conf.  =  10/10 


RULE  NUMBER:  11 
IF: 

the  flying  schedule  within  the  DB  is  different 
or  the  mission  types  within  the  scenario  to  be  used  are 
different 

THEN: 

Modify  the  scenario  to  fit  the  required  parameters. 
The  need  to  modify  the  DB  is  at-  Conf.  »  10/10 
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RULE  NUMBER:  12 
IF: 

the  DB  to  be  used  has  been  run  within  the  LOOM 
simulation  model 

THEN: 

Proceed  with  the  COEA  Information  Gathering  Process. 
The  DB  Integrity  Check  is  at  -  Conf.  =*  10/10 


RULE  NUMBER:  13 
IF: 

the  DB  has  been  run  within  the  LCOM  simulation  model, 
and  the  results  are  good 

THEN: 

The  COEA  Information  Gathering  Process  is  now 
complete.  Inform  the  COELA  expert  that  all  the 
required  information  is  now  in— hand  and  that  the  COEA 
Proposal  is  ready  to  be  run  -  Conf.  -  10/10 


RULE  NUMBER:  14 
IF; 

the  DB  to  be  used  has  not  been  run  within  the  LCOM 
simulation  model 

THEN : 

Run  the  DB  through  the  LCOM  simulation  model.  The 
need  for  a  DB  Integrity  Check  is  at-  Conf.  =  10/10 


RULE  NUMBER:  15 
IF: 

the  DB  has  been  run  within  the  LCOM  simulation  model, 
and  the  results  are  bad 

THEN: 

Investigate  the  errors  specified  from  the  DB 
Integrity  Check  by  the  LCOM  simulation  model . 
Determine  the  magnitude  of  the  corrections  necessary. 
The  need  to  bring  the  DB  up  to  the  necessary 
integrity  is  at  -  Conf.  *  10/10 
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RULES: 


II.  TAG  THUNDER  Prototype 


RULE  NUMBER:  1 
IF: 

the  scenario  year  is  consistent 

THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
year  of  study  consistency  is  at  -  Confidence-1 


RULE  NUMBER:  2 
IF: 

the  scenario  year  is  inconsistent 

THEN; 

Review  the  consistency  of  the  study  year  with  the 
study  leader.  Study  year  inconsistency  is  at  - 
Confidence®! 


RULE  NUMBER:  3 
IF: 

several  years  have  been  identified 

THEN : 

Proceed  with  COEA  Information  Gathering  Process.  The 
availability  of  the  necessary  niimber  of  study  years 
is  at  -  Confidence-1 


RULE  NUMBER:  4 
IF: 

several  years  have  not  been  identified 

THEN: 

Ask  study  leader  if  single  year  analysis  is 
acceptable.  Availability  of  only  a  single  study  year 
is  at  -  Confidence-1 


RULE  NUMBER:  5 
IF: 

the  scenarios  are  traceable  back  to  DPG/IPS 

THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
traceability  of  the  scenarios  is  at  -  Confidence-1 
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RULE  NUMBER:  6 
IF: 

the  scenarios  are  not  traceable  back  to  DPG/IPS 

THEN: 

Provide  rationale  from  user/sponsor  for  use  of 
non-DPG/IPS  scenarios.  Use  of  non-DPG/IPS  scenarios 
is  at  -  Confidence*! 


RULE  NUMBER:  7 
IF: 

the  scenarios  seem  appropriate 

THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
appropriateness  of  the  scenarios  used  is  at  - 
Confidence*! 


RULE  NUMBER:  8 
IF: 

the  scenarios  seem  contrived 

THEN: 

Review  scenario  problems  with  study  leader. 

Document /provide  rationale  for  questionable  areas  and 
get  study  leader  approval  to  continue  process.  - 
Confidence-1 


RULE  NUMBER:  9 
IF: 

the  scenarios  have  identified  the  mission  tasking  for 
the  alternatives 

THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
likelihood  that  the  scenarios  do  identify  tasking 
alternatives  is  at  -  Confidence-1 


RULE  NUMBER:  10 

the  scenarios  have  not  identified  the  mission  tasking 
for  the  alternatives 


Get  a  complete  listing  of  mission  taskings  from  the 
user/sponsor.  The  need  for  this  listing  is  at  - 
Confidence-1 


IF: 


THEN  : 
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RULE  NUMBER:  11 
IF: 

at  least  one  of  the  scenarios  provide (s)  a(n) 
stressful  and  likely  cases 

or:  at  least  one  of  the  scenarios  provide (s)  a(n) 

unlikely  case 

THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
likelihood  that  the  scenarios  provide  all  cases  is  at 
-  Confidence*! 

ELSE: 

Review  scenario  cases  with  the  study  leader.  The 
likelihood  of  a  problem  with  one  or  more  of  the 
scenarios  is  at  -  Confidence*! 


RULE  NUMBER:  12 
IF: 

the  scenarios  present  a(n)  good  operational  range  of 
possibilities 

THEN : 

Proceed  with  COEA  Information  Gathering  Process.  The 
proposed  scenarios  do  present  a  good  range  of 
possibilities  is  at  -  Confidence*! 


RULE  NUMBER:  13 
IF: 

the  scenarios  present  a(n)  unacceptable  range  of 
possibilities 

THEN: 

Consider  adding  additional  scenario (s)  to  get  a  good 
operational  range.  -  Confidence*! 


RULE  NUMBER:  14 
IF: 

the  number  of  theaters  selected  is  more  than  one 

THEN: 

Proceed  with  COEA  Information  Gathering  Process. 
Theater  selection  is  at  -  Confidence-1 


RULE  NUMBER:  15 
IF: 

the  number  of  theaters  selected  is  one 
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THEN: 


Get/provide  rationale  from  the  user/sponsor  for  a  one 
theater  option.  If  no  rationale  forthcoming,  get 
study  leader  approval  before  continuing  process.  - 
Confidence*! 


RULE  NUMBER:  16 
IF: 

the  DPG/IPS  scenarios  are  built  on  a  validated  threat 

THEN: 

Proceed  with  COEA  Information  Gathering  Process. 

Good  threat  assessment  is  at  -  Confidence-1 


RULE  NUMBER:  17 
IF: 

the  DPG/IPS  scenarios  are  built  on  a  nonvalidated 
threat 

THEN: 

Obtain  STAR  (System  Threat  Assessment  Report)  from 
FASTC.  The  need  for  this  information  is  at  - 
Confidence-1 


RULE  NUMBER:  18 
IF: 

blue  systems  aircraft  and  ground  assets  are  correctly 
modeled 

THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
correct  representation  of  the  blue  forces  is  at  - 
Confidence-1 


RULE  NUMBER :  1 9 
IF: 

blue  systems  aircraft  and  ground  assets  are 
incorrectly  modeled 

THEN; 

Contact  mission  level  office  for  a  review  of  bi  le 
system  assets.  -  Confidence-1 


RULE  NUMBER:  20 
IF: 

the  user/sponsor  has  reviewed  '■he  proposed  s  enar: 

database 


THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
need  for  user/sponsor  review  of  the  proposed  scenario 
is  at  --  Confidence-1 


RULE  NUMBER:  21 
IF: 

the  user/ sponsor  has  not  reviewed  the  proposed 
scenario  database 

THEN: 

Get  approval /coordination  from  study  leader  and 
user/sponsor .  Incorporate  any  changes  noted.  - 
Confidence-1 


RULE  NUMBER:  22 
IF: 

the  mission  tasks  for  the  system  have  been  identified 
based  on  need 

THEN  : 

Proceed  with  COEA  Information  Gathering  Process.  The 
identification  of  mission  tasks  based  upon  need  is  at 
-  Confidence-0 


RULE  NUMBER:  23 

IF: 

the  mission  tasks  for  the  system  have  not  been 
identified  based  on  need 

THEN: 

Coordinate  with  study  leader  to  formulate  appropriate 
tasks  and/or  study  measures  for  the  alternatives 
being  examined.  -  Confidence-1 


RULE  NUMBER;  24 

IF  : 

the  eission  t-asks  are  quantified 

THEN 

Proceed  with  COEA  Information  Gathering  Process.  The 
quantification  of  mission  tasks  is  at  -  Confidence-1 


RULE  NtJMBBR 
If 

»  he  missi'jn  ♦asks  are  no»  ^uanrified 


■  f. 


THEN: 

Develop  quantifiable  mission  objectives/tasks.  - 
Confidence*! 

RULE  NUMBER:  26 
IF: 

the  employment  of  the  system  is  feasible 

THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
feasibility  of  the  employment  portion  of  the  study  is 
at  -  Confidence*! 


RULE  NUMBER:  27 
IF: 

the  employment  of  the  system  is  not  feasible 

THEN: 

Review  employment  concept  with  user/sponsor.  - 
Confidence*! 


RULE  NUMBER:  28 
IF: 

the  operations  and  maintenance  force  structure  is 
valid 

THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
validity  of  the  overall  force  structure  is  at  - 
Confidence*! 


RULE  NUMBER:  29 
IF: 

the  operations  and  maintenance  force  structure  is  not 
valid 

THEN  ; 

Review  force  structure  with  user/sponsor.  - 
Confidence*! 


RULE  NUMBER:  30 
IF: 

the  interfaces  with  other  systems  have  been 
considered 

THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
consideration  of  other  interfaces  is  at  - 
Confidence-1 
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RULE  NUMBER:  31 
IF: 

the  interfaces  with  other  systems  have  not  been 
considered 


THEN: 

Obtain  interface  data  from  user/sponsor.  - 
Confidence*! 


RULE  NUMBER:  32 
IF: 

each  alternative  has  been  described  in  detail 


THEN: 

Proceed  with  COEA  Information  Gathering  Process. 
Each  alternative's  description  is  at  -  Confidence*! 


RULE  NUMBER:  33 
IF: 

each  alternative  has  not  been  described  in  detail 


THEN: 

Obtain  necessary  description  details  from 
user/sponsor.  -  Confidence*! 


RULE  NUMBER:  34 
IF: 

the  current  system  baseline  has  been  identified 


THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
baseline  case  identification  is  at  -  Confidence*! 


RULE  NUMBER:  35 
IF: 

the  current  system  baseline  has  not  been  identified 

THEN: 

Review  baseline  case  with  user/sponsor.  - 
Confidence*! 


RULE  NUMBER:  36 

IF: 

the  adequate  range  of  alternatives  has  been 
identified 


'8 


THEN: 


Proceed  with  COEA  Information  Gathering  Process.  The 
availability  of  an  adequate  range  is  at  - 
Confidence*! 


RULE  NUMBER:  37 
IF: 

the  adequate  range  of  alternatives  has  not  been 
identified 

THEN: 

Review  proposed  range  with  study  leader  and 
user/sponsor .  Provide  rationale  for  proposed  range. 
-  Confidence*! 


RULE  NUMBER:  38 
IF: 

the  alternatives  do  consider  changes  in  the 
requirements 

THEN: 

Proceed  with  COEA  Information  Gathering  Process.  The 
adequacy  of  the  changes  considered  is  at  - 
Confidence*! 


RULE  NUMBER:  39 
IF: 

the  alternatives  do  not  consider  changes  in  the 
requirements 

THEN: 

Review  with  study  leader  and  user/sponsor.  Provide 
rationale  for  this  consideration  of  requirements.  - 
Confidence*! 


RULE  NUMBER:  40 
IF: 

the  current  or  prospective  systems  contain  reasonable 
alternatives 


Proceed  with  rOEA  Information  Gathering  Process 
reasonableness  of  the  alternatives  i s  at  - 
Confidence- 1 


■  4 


THEN; 


The 


RULE  NUMBER:  41 
IF: 

the  current  or  prospective  systems  contain 
unreasonable  alternatives 

THEN: 

Review  with  study  leader  and  user/sponsor.  Document 
rationale  for  using  specified  alternatives.  - 
Confidence*! 


RULE  NUMBER:  42 
IF: 

the  rationale  for  the  non-selection  for  alternatives 
has  been  explained 

THEN : 

Proceed  with  COEA  Information  Gathering  Process . 
Non-selection  rationale  availability  is  at  - 
Confidence*! 


RULE  NUMBER;  43 
IF: 

the  rationale  for  the  non-selection  for  alternatives 
has  not  been  explained 

THEN  : 

Document  rationale  for  non-selection  of  alternatives. 
-  Confidence*! 


III.  Gatherer  Prototype 

RULES: 


RULE  NUMBER:  1  (GENERAL  RQMTS — 2) 

IF: 

The  scenario  is  a  _  format:  Peacetime 

THEN: 

Scenario  is  Peacetime  -  Confidence-1 


RULE  NUMBER:  2  (GENERAL  RQMTS — 3) 

IF: 

The  scenario  is  a  _  format:  Wartime 

THEN: 

Scenario  is  Wartime  -  Confidence-1 


RULE  NUMBER:  3  (GENERAL  RQMTS — 4) 

IF: 

The  scenario  is  a  _  format:  other 

THEN: 

LCOM  and  TAC  THUNDER  are  not  designed  for  scenarios 
other  than  Peacetime  or  Wartime  -  Confidence-1 
and  See  COEA  focal  point  for  further  guidance  - 
Confidence-1 


RULE  NUMBER:  4  (GENERAL  RC^S — 5) 

IF: 

The  scenario/IOC  year  consistency  is:  No 

THEN; 

Review  scenario/ ICX:  year  with  user  to  resolve  and/or 
verify  discrepancy  -  Confidence-1 

ELSE  ; 

The  scenario/IOC  year  is:  -  Confidence-1 


RULE  NUMBER  I  GENERAL  RQMTS  —  h) 

ir 

The  Engineer inq  Change  Proposal  is  not  available 

'hecli  valid'ty  >f  Engineering  hange  Proposa. 
user  ■'  n  f  I  den<  e- 1 


then 


I »  h 


ELSE: 

The  Engineering  Change  Proposal  information  is:  - 
Confidence-1 

RULE  NUMBER:  6  (F&D — 2) 

IF: 

The  number  of  locations  and  number  of  aircraft  at 
each  site  have:  not  been  determined 

THEN: 

Get  the  numbers  and  locations  of  aircraft  at  each 
site  from  users  -  Confidence-1 

ELSE: 

The  numbers  and  locations  of  aircraft  at  each  site 
are :  -  Confidence-1 


RULE  NUMBER:  7  (FSD — 3) 

IF: 

The  supply  concept  (for  example : deploy  with  War 
Readiness  Spares  Kit  for  xxx  days)  has:  not  been 
determined 

THEN: 

Get  the  supply  concept  information  from  user  - 
Confidence-1 

ELSE : 

The  supply  concept  is:  -  Confidence-1 


RULE  NUMBER:  8  (F&D — 4) 

IF: 

The  resupply  time  has:  not  been  determined 

THEN: 

Get  the  resupply  time  information  from  user  - 
Confidence-1 

ELSE; 

The  supply  concept  is:  -  Confidence-1 


RULE  NUMBER  9  (F&D — 5) 

IF 

The  extent  of  maintenance  capability  required  at  each 
site  has:  not  been  determined 

THEN 

dmt  the  extent  of  maintenance  capability  required  at 
each  site  from  user  -  Confidence-1 


ELSE: 

The  maintenance  capability  required  at  each  site  is: 

-  Confidence-1 

RULE  NUMBER:  10  (F&D— 6) 

IF: 

The  shelters  at  each  site  has:  not  been  determined 

THEN: 

Get  the  shelter  information  for  each  site  from  user  - 
Confidence-1 

ELSE: 

The  shelter  information  at  each  site  is:  - 
Confidence-1 


RULE  NUMBER:  11  (F&D — 7) 

IF: 

The  facilities  and  support  equipment  for  each  site 
have:  not  been  determined 

THEN: 

Get  the  facilities  and  support  equipment  requirements 
for  each  site  from  user  -  Confidence-1 

ELSE: 

The  facilities  and  support  equipment  requirements  for 
each  site  are:  -  Confidence-1 


RULE  NUMBER:  12  (MR — 2) 

IF: 

The  mission  types  (for  example:  Interdiction;  Combat 
Air  Patrol)  have:  not  been  determined 

THEN: 

Get  the  mission  types  from  user  -  Confidence-1 

ELSE: 

The  mission  types  are:  -  Confidence-1 


RULE  NUMBER:  13  (MR — 3) 

IF: 

The  aircraft  configuration  for  each  mission  has:  not 
been  determined 

THEN: 

Get  the  aircraft  configuration  for  each  mission  from 
user  -  Confidence-1 


ELSE : 

The  aircraft  configuration  for  each  mission  is:  - 
Confidence^! 

RULE  NUMBER:  14  (MR — 4) 

IF: 

The  mission  priorities  have:  not  been  determined 

THEN: 

Get  the  user  to  establish  mission  priorities  - 
Confidence**! 

ELSE: 

The  mission  priorities  are:  -  Confidence**! 


RULE  NUMBER:  15 '  (MR — 5) 

IF: 

The  mission  cancellation  delay  times  tolerances  have 
not  been  established 

THEN: 

Get  user  to  determine  mission  delay  time  tolerances  - 
Confidence-*! 

ELSE: 

The  tolerances  for  mission  delay  time  are:  - 
Confidence-*! 


RULE  NUMBER:  16  (MR — 6) 

IF: 

The  mission  tasks  for  the  system  (based  on  need) 
have:  not  been  determined 

THEN: 

Coordinate  with  user  to  formulate  tasks  and/or  study 
measures  for  alternatives  being  exaunined  - 
Confidence*! 

ELSE : 

The  mission  tasks  for  the  system  are:  -  Confidence*-! 


RULE  NUMBER:  17  (04S — 2) 

IF: 

The  aircraft  sortie  rates  have  not  been  determined 

THEN: 

Get  aircraft  sortie  rates  from  user  -  Confidence-1 

ELSE; 

The  aircraft  sortie  rates  are:  -  Confidence-1 

B4 
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RULE  NUMBER:  18  (O&S — 3) 

IF: 

The  requirements  for  complementary  missions  have:  not 
been  determined 

THEN: 

Get  user  to  determine  requirements  for 
complementary  missions  -  Confidence™! 

ELSE: 

The  requirements  for  complementary  missions  are:  - 
Confidence™! 


RULE  NUMBER:  19  (O&S — 4) 

IF: 

The  interfaces  with  other  systems  (for  example: 
support  aircraft  or  resources  for  particular 
missions)  have:  not  been  considered 

THEN: 

Review  the  lack  of  interface  data  with  user  - 
Confidence™! 


RULE  NUMBER:  20  (O&S — 5) 

IF: 

The  interfaces  with  other  systems  (for  example: 
support  aircraft  or  resources  for  particular 
missions)  have:  been  considered 

and  The  appropriate  interface  data  is:  available 

THEN: 

The  interfaces  with  other  systems  and  their  data  are: 
-  Confidence™! 


RULE  NUMBER:  21 
IF: 

The  interfaces  with  other  systems  (for  example ; 
support  aircraft  or  resources  for  particular 
missions)  have:  been  considered 

and  The  appropriate  interface  data  is.  not  available 

THEN: 

Get  data  for  interface  systems  from  user  - 
Conf idence-1 


RULE  NUMBER:  22  (GA — 2) 

IF: 

The  niiinber  of  aircraft  on  alert  at  each  location  is: 
not  available 

THEN: 

Get  number  of  aircraft  on  alert  at  each  location  from 
user  -  Confidence-1 

ELSE: 

The  number  of  aircraft  on  alert  at  each  location  is: 

-  Confidence-1 


RULE  NUMBER:  23  (GA — 3) 

IF: 

The  missions  to  be  flown  from  alert  have:  not  been 
determined 

THEN: 

Get  missions  to  be  flown  from  alert  information  from 
user  -  Confidence-1 

ELSE: 

Missions  to  be  flown  from  alert  are:  -  Confidence-1 


RULE  NUMBER:  24  (GA — 4) 

IF: 

The  frequency  of  alert  missions  has:  not  been 
determined 

THEN: 

Get  frequency  of  alert  missions  information  from  user 
-  Confidence-1 

ELSE  : 

The  frequency  of  alert  missions  is:  -  Confidence-1 

RULE  NUMBER:  25  (GA — 5) 

ir  - 

The  alert  replacement  policies  (for  example, 
replacement  when  launched  or  same  aircraft  return 
from  alert)  have  not  beer,  determined 

THEN 

vie*’  alert  replacement  policy  information  from  user  - 
'on  f  i  den  v  #-  1 

Kl.SE 

The  a.er*  tep.acemen*  po  .  .  >  .  !»  onfidence-l 


RULE  NUMBER:  26  (MC&O — 2) 

IF: 

The  maintenance  concept  (for  example:  2  level — remove 
and  replace;  3  level — repair  and  replace)  have:  not 
been  determined 

THEN: 

Get  user  to  determine  maintenance  concept  to  be  used 
-  Confidence-1 

ELSE: 

The  maintenance  concept  to  be  used  is:  -  Confidence-1 


RULE  NUMBER:  27  (MC&O — 3) 

IF: 

The  organizational  structure  and  maintenance  concept 
level:  does  not  match 

THEN  : 

Get  with  user  to  rectify  organization  structure  and 
maintenance  concept  level  -  Confidence-1 

ELSE: 

The  organization  structure  is:  -  Confidence-1 


RULE  NUMBER:  28  (MC&O — 4) 

IF: 

The  AFSC  structure  and  organizational  structure  are: 
not  in  compliance 

THEN : 

Get  with  user  to  rectify  differences  in  AFSC 
structure,  organization  structure,  and  maintenance 
concept  level-  Confidence-1 

ELSE: 

The  AFSC  structure  is:  -  Confidence-1 


RULE  NUMBER:  29  (MC&O — 5) 

IF: 

The  AFSC  structure  and  maintenance  concept  level 
specifications  are:  not  in  compliance 

THEN: 

Get  with  user  to  rectify  differences  in  AFSC 
structure,  organization  structure,  and  maintenance 
concept  level  -  Confidence-1 

ELSE: 

The  AFSC  structure  is:  -  Confidence-1 
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RULE  NUMBER:  30  (GENERAL  RMTS — 1) 

IF: 

The  scenario  is  a  _  format:  NOT  other 

and  The  scenario/IOC  year  consistency  is:  Yes 
and  The  Engineering  Change  Proposal  is:  available 

THEN: 

(jeneral  Requirements  -  Confidence-1 

ELSE: 

General  Requirements  -  Confidence-0 

RULE  NUMBER:  31  (F&O — 1) 

IF: 

The  number  of  locations  and  number  of  aircraft  at 
each  site  have:  been  determined 

and  The  supply  concept  (for  example : deploy  with  War 
Readiness  Spares  Kit  for  xxx  days)  has:  been 
determined 

and  The  resupply  time  has:  been  determined 

and  The  extent  of  maintenance  capability  required  at  each 
site  has:  been  determined 

and  The  shelters  at  each  site  has:  been  determined 

and  The  facilities  and  support  equipment  for  each  site 
have:  been  determined 

THEN: 

Facilities  and  Deployment  -  Confidence-1 

ELSE  : 

Facilities  and  Deployment  -  Confidence-0 

RULE  NUMBER:  32  (GA — 1) 

IF: 

The  number  of  aircraft  on  alert  at  each  location 
is:  available 

and  The  missions  to  be  flown  from  alert  have:  been 
determined 

and  The  frequency  of  alert  missions  has:  been  determined 

and  The  alert  replacement  policies  (for  excunple, 

replacement  when  launched  or  same  aircraft  return 
from  alert)  have:  been  determined 


88 


THEN: 

Ground  Alert  -  Confidence-1 

« 

ELSE: 

Ground  Alert  -  Confidence-0 

RULE  NUMBER:  33  (MC&O — 1) 

IF: 

The  maintenance  concept  (for  example:  2  level — remove 
and  replace;  3  level — repair  and  replace)  have:  been 
determined 

and  The  organizational  structure  and  maintenance 
concept  level:  matches 

and  The  AFSC  structure  and  organizational  structure  are: 
in  compliance 

and  The  AFSC  structure  and  maintenance  concept  level 
specifications  are:  in  compliance 

THEN: 

Maintenance  Concepts  and  Organization  -  Confidence-1 

ELSE: 

Maintenance  Concepts  and  Organization  -  Confidence-0 


RULE  NUMBER:  34  (MR — 1) 

IF: 

The  mission  types  (for  example:  Interdiction;  Combat 
Air  Patrol)  have:  been  determined 

and  The  aircraft  configuration  for  each  mission  has:  been 
determined 

and  The  mission  priorities  have:  been  determined 

and  The  mission  cancellation  delay  times  tolerances  have: 
been  established 

and  The  mission  tas}cs  for  the  system  (based  on  need) 
have:  been  determined 


THEN; 


Mission  Requirements  -  Confidence-1 


ELSE: 


Mission  Requirements  -  Confidence-0 
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RULE  NUMBER:  35  (O&S — 1) 

IF: 

The  aircraft  sortie  rates  have  been  determined 

and  The  requirements  for  complementary  missions  have: 
been  determined 

and  The  interfaces  with  other  systems  (for  example: 
support  aircraft  or  resources  for  particular 
missions)  have:  been  considered 

and  The  appropriate  interface  data  is:  available 

THEN: 

Operations  and  Scheduling  Policy  -  Confidence-1 

ELSE: 

Operations  and  Scheduling  Policy  -  Confidence-0 


RULE  NUMBER:  36 
IF: 

General  Requirements-  Conf.  -  1 
and  Facilities  and  Deployment-  Conf.  -  1 
and  Mission  Requirements-  Conf.  -  1 
and  Operations  and  Scheduling  Policy-  Conf.  -  1 
and  Ground  Alert-  Conf.  -  1 

and  Maintenance  Concepts  and  Organization-  Conf.  »  1 
THEN: 

Proceed  to  disseminate  information  for  COEA  Request 
to  LCOM  and  TAC  THUNDER.  -  Confidence-1 

ELSE : 

Not  all  information  is  available  for  LCOM  and  TAC 
THUNDER  to  process  this  COEA  -  Confidence-1 
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Appendix  F:  User  Interface  Messages 


I.  LOOM 


Starting  text: 

Welcome  to  the  Cost  and  Operational  Effectiveness  Analysis 
(COEA)  Information  Gathering  Process  Expert  System  for  the 
Logistics  Composite  Model  (LCOM) .  If  at  anytime  you  do  not 
understand  what  any  of  the  questions  are  asking  for,  please 
consult  the  accompanying  doc\imentation  or  as  a  last  resort 
the  COEA  expert  within  the  LCOM  area.  If  there  are 
questions  that  are  not  covered  by  any  of  the  above  sources, 
please  consult  with  the  expert  system  designers  Constance  S. 
Maginnis  or  Michael  J.  Monroe  at  255-8989. 

Ending  text: 

Thank  you  for  using  the  LCOM  COEA  Information  Gathering 
Process  Expert  System.  The  next  screen  will  display  the 
results  of  the  current  Data  Run.  Each  area  of  query 
answered  and  the  confidence  level  is  listed  separately  in 
order  of  the  questions  asked.  You  may  change  any  of  the 
initial  parameters  by  clicking  on  the  <Change/Rerun>  button 
and  modifying  the  parameter (s)  desired.  The  latest  Data  Run 
will  appear  in  the  first  column  and  the  initial  Data  Run 
will  appear  in  the  second  column.  You  can  easily  compare 
the  impact  of  one  alteration  of  the  input  parameters. 


II.  TAC  THUNDER 


Starting  text: 

Welcome  to  the  Cost  and  Operational  Effectiveness  Analysis 
(COEA)  Information  Gathering  Process  Expert  System  for  the 
TAC  THUNDER  model.  If  at  anytime  you  do  not  understand  what 
any  of  the  questions  are  asking  for,  please  consult  the 
accompanying  documentation  or  as  a  last  resort  the  COEA 
expert  within  the  TAC  THUNDER  area.  If  there  are  questions 
that  are  not  covered  by  any  of  the  above  sources,  please 
consult  with  the  expert  system  designers  Constance  S. 
Maginnis  or  Michael  J.  Monroe  at  255-8989. 

Ending  text: 

Thank  you  for  using  the  TAC  THUNDER  COEA  Information 
Gathering  Process  Expert  System.  The  next  screen  will 
display  the  results  of  the  current  Data  Run.  Each  area  of 
query  answered  and  the  confidence  level  is  listed  separately 
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in  order  of  the  questions  asked.  You  may  change  any  of  the 
initial  parameters  by  clicking  on  the  <Change/Rerun>  button 
and  modifying  the  parameter (s)  desired.  The  latest  Data  Run 
will  appear  in  the  first  column  and  the  initial  Data  Run 
will  appear  in  the  second  colximn.  You  can  easily  compare 
the  impact  of  one  alteration  of  the  input  parameters. 


III.  COEA  Gatherer 


S^artin9  Text: 

Welcome  to  the  Cost  and  Operational  Effectiveness  Analysis 
(COEA)  Information  Gathering  Process  Expert  System.  If  at 
anytime  you  do  not  understand  what  any  of  the  questions  are 
asking  for,  please  consult  the  accompanying  documentation  or 
as  a  last  resort  the  COEA  expert  within  the  Gatherer  area. 

If  there  are  questions  that  are  not  covered  by  any  of  the 
above  sources,  please  consult  with  the  expert  system 
designers  Constance  S.  Maginnis  or  Michael  J.  Monroe  at 
255-8989. 


Ending  Text: 

Thank  you  for  using  the  COEA  Gatherer.  The  next  screen  will 
display  the  results  of  the  current  Data  Run.  Each  area  of 
query  answered  and  the  confidence  level  is  listed  separately 
in  order  of  the  questions  asked.  There  are  several 
different  files  that  can  be  saved  and  printed  for  the  user' s 
convenience.  Please  see  the  accompanying  documentation  for 
full  details. 
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Appendix  G:  Knowledge  Base 


I.  LOOM 


QOALirilRS: 


1  the  Engineering  Change  Proposal  provided  with  the  COEA  is 

available 
not  available 

2  the  Logistics  Support  Analysis  aircraft  data  is 

available 
not  available 

3  other  Air  Force  maintenance  data  is 

available 
not  available 

4  a  historical  Air  Force  Maintenance  Database  (DB)  is 

available 
not  available 

5  other  Non-AF  DB  are 

available 
not  available 

6  the  sequence  of  maintenance  tasks  to  be  performed  is 

available 
not  available 

7  the  resource  requirements  for  each  specific  maintenance 
task  are 

available 
not  available 

8  the  task  time  for  each  maintenance  task  to  be  performed 
is 

available 
not  available 
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♦ 

9  the  specifications  for  each  maintenance  crew  required  to 
perform  each  maintenance  task  are 

available 
not  available 

10  the  support  facilities  required  for  each  maintenance  task 
to  be  performed  are 

available 
not  available 

11  the  reliability  values  required  for  each  maintenance  task 
to  be  performed  are 

available 
not  available 

12  the  specific  LOOM  DB  scenario  is 

available  in-house 
not  available  in-house 

13  the  flying  schedule  within  the  DB  is 

the  same 
different 

14  the  mission  types  within  the  scenario  to  be  used  are 

the  same 
different 

15  the  DB  to  be  used  has 

been  run  within  the  LOOM  simulation  model 
not  been  run  within  the  LCOM  simulation  model 

16  the  DB  has  been  run  within  the  LCOM  simulation  model,  and 
the  results  are 

good 

bad 


CHOICES : 

1  Proceed  with  the  COEA  Information  Gathering  Process.  ECP 
availability  is  at 

2  Check  the  validity  of  the  Proposed  COEA  with  the  Systems 
Program  Office  and  ,  if  necessary,  consult  with  the  COEA 
expert.  ECP  nonavailability  is  at 
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3  Proceed  with  the  COEA  Information  Gathering  Process.  The 
necessary  Analysis  Data  availability  is  at 

4  Check  the  validity  of  the  COEA  Proposal  with  the  Systems 
Program  Office  and  the  COEA  expert  since  there  is  no 
Analysis  Data  available  at 

5  Continue  the  COEA  Information  Gathering  Process.  The 
Specific  Task  Data  availability  is  at 

6  Look  for  comparable  data  that  matches  the  Specified  Task 
Data  or  ask  the  COEA  expert  for  guidance  in  finding  data 
for  that  specific  task  area.  The  nonavailability  of  the 
Specific  Task  Data  is  at 

7  Proceed  with  the  COEA  Information  Gathering  Process.  The 
Scenario  availability  is  at 

8  Check  other  sources  with  the  same  type  of  aircraft  as 
indicated  in  the  COEA  Proposal  (like  MAJCOMs,  other 
services,  or  commercial  aviation  services)  to  get  the 
needed  Scenario  specific  data.  The  need  for  Scenario 
specific  data  is  at 

9  Ensure  the  validity  of  the  proposed  COEA  with  the  Systems 
Program  Office  and  the  COEA  expert.  The  nonavailability 
of  the  specific  Scenario  is  at 

10  Continue  with  the  COEA  Information  Gathering  Process. 

The  availability  of  the  DB  is  at 

11  Modify  the  scenario  to  fit  the  required  parameters.  The 
need  to  modify  the  DB  is  at 

12  Proceed  with  the  COEA  Information  Gathering  Process.  The 

•  DB  Integrity  Check  is  at 

13  The  COEA  Information  Gathering  Process  is  now  complete. 
Inform  the  COEA  expert  that  all  the  required  information 
is  now  in-hand  and  that  the  COEA  Proposal  is  ready  to  be 
run. 

14  Run  the  DB  through  the  LOOM  simulation  model.  The  need 
for  a  DB  Integrity  Check  is  at 

15  Investigate  the  errors  specified  from  the  DB  Integrity 
Check  by  the  LCOM  simulation  model.  Determine  the 
magnitude  of  the  corrections  necessary.  The  need  to 
bring  the  DB  up  to  the  necessary  integrity  is  at 
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II.  TAG  THUNDER  COEA  KBS 


QCALIFIBRS: 


1  the  scenario  year  is 

consistent 

inconsistent 

2  several  years  have 

been  identified 
not  been  identified 

3  the  scenarios  are 

traceable  back  to  DPG/IPS 
not  traceable  back  to  DPG/IPS 

4  the  scenarios  seem 

appropriate 

contrived 

5  the  scenarios  have 

identified  the  mission  tasking  for  the  alternatives 
not  identified  the  mission  tasking  for  the  alternatives 

6  at  least  one  of  the  scenarios  provide (s)  a(n) 
unlikely 

stressful  and  likely  cases 

7  the  scenarios  present  a (n) 

good  operational  range  of  possibilities 
unacceptable  range  of  possibilities 

8  the  number  of  theaters  selected  is 
one 

more  than  one 

9  the  DPG/IPS  scenarios  are  built  on  a 

validated  threat 
nonvalidated  threat 
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% 

10  blue  systems  aircraft  and  ground  assets  are 

correctly  modeled 
incorrectly  modeled 

11  the  user/sponsor  has 

reviewed  the  proposed  scenario  database 
not  reviewed  the  proposed  scenario  database 

12  the  mission  tasks  for  the  system  have 

not  been  identified  based  on  need 
been  identified  based  on  need 

13  the  mission  tasks  are 

quantified 
not  quantified 

14  the  employment  of  the  system  is 

feasible 
not  feasible 

15  the  operations  and  maintenance  force  structure  is 

valid 
not  valid 

16  the  interfaces  with  other  systems  have 

been  considered 
not  been  considered 

17  each  alternative  has 

been  described  in  detail 
not  been  described  in  detail 

18  the  current  system  baseline  has 

been  identified 
not  been  identified 

19  the  adequate  range  of  alternatives  has 

been  identified 
not  been  identified 
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♦ 

20  the  alternatives  do 

consider  changes  in  the  requirements 
not  consider  changes  in  the  requirements 

21  the  current  or  prospective  systems  contain 

reasonable  alternatives 
unreasonable  alternatives 

22  the  rationale  for  the  non-selection  for  alternatives  has 

been  explained 
not  been  explained 


CHOICES: 


1  Proceed  with  COEA  Information  Gathering  Process.  The 
year  of  study  consistency  is  at 

2  Review  the  consistency  of  the  study  year  with  the  study 
leader.  Study  year  inconsistency  is  at 

3  Proceed  with  COEA  Information  Gathering  Process.  The 
availability  of  the  necessary  number  of  study  years  is  at 

4  Ask  study  leader  if  single  year  analysis  is  acceptable. 
Availability  of  only  a  single  study  year  is  at 

5  Proceed  with  COEA  Information  Gathering  Process.  The 
traceability  of  the  scenarios  is  at 

6  Provide  rationale  from  user/sponsor  for  use  of 
non-DPG/IPS  scenarios.  Use  of  non-DPG/IPS  scenarios  is  at 

7  Proceed  with  COEA  Information  Gathering  Process.  The 
appropriateness  of  the  scenarios  used  is  at 

8  Review  scenario  problems  with  study  leader. 
Document/provide  rationale  for  questionable  areas  and  get 
study  leader  approval  to  continue  process. 

9  Proceed  with  COEA  Information  Gathering  Process.  The 
likelihood  that  the  scenarios  do  identify  tasking 
alternatives  is  at 

10  Get  a  complete  listing  of  mission  taskings  from  the 
user/sponsor.  The  need  for  this  listing  is  at 

11  Proceed  with  COEA  Information  Gathering  Process.  The 
likelihood  that  the  scenarios  provide  all  cases  is  at 
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12  Review  scenario  cases  with  the  study  leader.  The 
likelihood  of  a  problem  with  one  or  more  of  the  scenarios  is 
at 

13  Proceed  with  COEA  Information  Gathering  Process.  The 
proposed  scenarios  do  present  a  good  range  of  possibilities 
is  at 

14  Consider  adding  additional  scenario (s)  to  get  a  good 
operational  range . 

15  Proceed  with  COEA  Information  Gathering  Process.  Theater 
selection  is  at 

16  Get/provide  rationale  from  the  user/sponsor  for  a  one 
theater  option.  If  no  rationale  forthcoming,  get  study 
leader  approval  before  continuing  process. 

17  Proceed  with  COEA  Information  Gathering  Process.  Good 
threat  assessment  is  at 

18  Obtain  STAR  (System  Threat  Assessment  Report)  from  FASTC . 
The  need  for  this  information  is  at 

19  Proceed  with  COEA  Information  Gathering  Process.  The 
correct  representation  of  the  blue  forces  is  at 

20  Contact  mission  level  office  for  a  review  of  blue  system 
assets . 

21  Proceed  with  COEA  Information  Gathering  Process.  The 
need  for  user/sponsor  review  of  the  proposed  scenario  is  at 

22  Get  approval /coordination  from  study  leader  and 
user/sponsor.  Incorporate  any  changes  noted. 

23  Proceed  with  COEA  Information  Gathering  Process.  The 
identification  of  mission  tasks  based  upon  need  is  at 

24  Coordinate  with  study  leader  to  formulate  appropriate 
tasks  and/or  study  measures  for  the  alternatives  being 
examined. 

25  Proceed  with  COEA  Information  Gathering  Process.  The 
quantification  of  mission  tasks  is  at 

26  Develop  quantifiable  mission  objectives/tasks. 

27  Proceed  with  COEA  Information  Gathering  Process.  The 
feasibility  of  the  employment  portion  of  the  study  is  at 

28  Review  employment  concept  with  user/sponsor. 
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29  Proceed  with  COEA  Information  Gathering  Process.  The 
validity  of  the  overall  force  structure  is  at 

30  Review  force  structure  with  user/sponsor. 

31  Proceed  with  COEA  Information  Gathering  Process.  The 
consideration  of  other  interfaces  is  at 

32  Review  the  lack  of  system  interface  data  with 
user/sponsor. 

33  Proceed  with  COEA  Information  Gathering  Process.  The 
appropriate  interface  data  availability  is  at 

34  Obtain  interface  data  from  user/sponsor . 

35  Proceed  with  COEA  Information  Gathering  Process.  Each 
alternative' s  description  is  at 

36  Obtain  necessary  description  details  from  user/sponsor. 

37  Proceed  with  COEA  Information  Gathering  Process.  The 
baseline  case  identification  is  at 

38  Review  baseline  case  with  user/sponsor . 

39  Proceed  with  COEA  Information  Gathering  Process.  The 
availability  of  an  adequate  range  is  at 

40  Review  proposed  range  with  study  leader  and  user/ sponsor . 
Provide  rationale  for  proposed  range. 

41  Proceed  with  COEA  Information  Gathering  Process.  The 
adequacy  of  the  changes  considered  is  at 

42  Review  with  study  leader  and  user/sponsor.  Provide 
rationale  for  this  consideration  of  requirements. 

43  Proceed  with  COEA  Information  Gathering  Process.  The 
reasonableness  of  the  alternatives  is  at 

44  Review  with  study  leader  and  user/sponsor.  Document 
rationale  for  using  specified  alternatives. 

45  Proceed  with  COEA  Information  Gathering  Process. 
Non-selection  rationale  availeUDility  is  at 

46  Document  rationale  for  non-selection  of  alternatives. 


f 
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III.  Gatherer 


QOMLIWIEBS: 

1  The  scenario  is  a  _  format: 

Peacetime 

Wartime 

other 

Name:  GENERAL  RQMTS — 1 
Maximum  acceptable  -  1 

2  The  scenario/IOC  year  consistency  is: 

Yes 

No 

Name:  GENERAL  RQMTS — 2 
Maximtim  acceptable  -  1 

3  The  Engineering  Change  Proposal  is: 

available 
not  available 

Name:  GENERAL  RQMTS — 3 
Maximum  acceptable  -  1 

4  The  number  of  locations  and  number  of  aircraft  at  each 
site  have: 

been  determined 
not  been  determined 

Name:  F  &  D — 1 
Maximum  acceptable  -  1 

5  The  supply  concept  (for  example: deploy  with  War  Readiness 
Spares  Kit  for  xxx  days)  has: 

been  determined 
not  been  determined 

Name:  F  &  D — 2 
Maximvim  acceptable  -  1 
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6  The  resupply  time  has: 

been  determined 
not  been  determined 

Name:  F  &  D — 3 
Maximum  acceptable  »  1 

7  The  extent  of  maintenance  capability  required  at  each 
site  has: 

been  determined 
not  been  determined 

Name:  F  &  D — 4 
Maximum  acceptable  »  1 

8  The  shelters  at  each  site  has: 

been  determined 
not  been  determined 

Name:  F  &  D — 5 
Maximum  acceptable  =>  1 

9  The  facilities  and  support  equipment  for  each  site  have: 

been  determined 
not  been  determined 

Name:  F  &  D — 6 
Maximum  acceptable  »  1 

10  The  mission  types  (for  example:  Interdiction;  Combat  Air 
Patrol)  have: 

been  determined 
not  been  determined 

Name :  MR — 1 

Maximum  acceptable  »  1 

11  The  aircraft  configuration  for  each  mission  has: 

been  determined 
not  been  determined 

Name:  MR — 2 

Maximvim  acceptable  »  1 
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12  The  mission  priorities  have: 

been  determined 
not  been  determined 

Name :  MR — 3 

Maximum  acceptable  =•  1 

13  The  mission  cancellation  delay  times  tolerances  have: 

been  established 
not  been  established 

Name :  MR — 4 

Maximum  acceptable  »  1 

14  The  mission  tasks  for  the  system  (based  on  need)  have: 

been  determined 
not  been  determined 

Name :  MR — 5 

Maximum  acceptable  >  1 

15  The  aircraft  sortie  rates  have 

been  determined 
not  been  determined 

Name:  O&SP — 1 
Maximum  acceptable  »  1 

16  The  requirements  for  complementary  missions  have: 

been  determined 
not  been  determined 

Name:  OiSP — 2 
Maximum  acceptable  >=  1 

17  The  interfaces  with  other  systems  (for  example:  support 
aircraft  or  resources  for  particular  missions)  have: 

been  considered 
not  been  considered 

Name:  OiSP — 3 
Maximum  acceptable  »  1 
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18  The  appropriate  interface  data  is: 


available 
not  available 

Name:  O&SP — 4 
Maximum  acceptable  -  1 

19  The  number  of  aircraft  on  alert  at  each  location  is: 

available 
not  available 

Ncune :  GA — 1 

Maximxim  acceptable  -  1 

20  The  missions  to  be  flown  from  alert  have: 

been  determined 
not  been  determined 

Name :  GA — 2 

Maximiim  acceptable  »  1 

21  The  frequency  of  alert  missions  has: 

been  determined 
not  been  determined 

Name :  GA — 3 

Maximum  acceptable  -  1 

22  The  alert  replacement  policies  (for  example,  replacement 
when  launched  or  same  aircraft  return  from  alert)  have: 

been  determined 
not  been  determined 

Name :  GA — 4 

Maximum  acceptable  »  1 

23  The  maintenance  concept  (for  example:  2  level — remove  and 
replace;  3  level — repair  and  replace)  have: 

been  determined 
not  been  determined 

Name :  MC&O — 1 
Maximum  acceptable  >  1 
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24  The  organizational  structure  and  maintenance  concept 
level : 

matches 

does  not  match 

Name :  MC&O — 2 
Maximum  acceptable  -  1 

25  The  AFSC  structure  and  organizational  structure  are: 

in  compliance 
not  in  compliance 

Name:  MC&O — 3 
Maximum  acceptable  -  1 

26  The  AFSC  structure  and  maintenance  concept  level 
specifications  are: 

in  compliance 
not  in  compliance 

Name :  MC&O — 4 
Maximum  acceptable  *  1 


CHOZdS: 

1  Proceed  to  disseminate  information  for  COEA  Request  to 
LOOM  and  TAC  THUNDER. 

2  Not  all  information  is  available  for  LOOM  and  TAC  THUNDER 
to  process  this  COEA 

3  General  Requirements 

4  Facilities  and  Deployment 

5  Mission  Requirements 

6  Operations  and  Scheduling  Policy 

7  Ground  Alert 

8  Maintenance  Concepts  and  Organization 

9  Scenario  is  Peacetime 

10  Scenario  is  Wartime 

11  LOOM  and  TAC  THUNDER  are  not  designed  for  scenarios  other 
than  Peacetime  or  Wartime 
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12  See  COEA  focal  point  for  further  guidance 

13  Review  scenario/ IOC  year  with  user  to  resolve  and/or 
verify  discrepancy 

14  The  scenario/IOC  year  is: 

15  Check  validity  of  Engineering  Change  Proposal  with  user 

16  The  Engineering  Change  Proposal  information  is: 

17  Get  the  numbers  and  locations  of  aircraft  at  each  site 
from  users 

18  The  numbers  and  locations  of  aircraft  at  each  site  are: 

19  Get  the  supply  concept  information  from  user 

20  The  supply  concept  is: 

21  Get  the  resupply  time  information  from  user 

22  The  resupply  time  is: 

23  Get  the  extent  of  maintenance  capability  required  at  each 
site  from  user 

24  The  maintenance  capability  required  at  each  site  is: 

25  Get  the  shelter  information  for  each  site  from  user 

26  The  shelter  information  at  each  site  is: 

27  Get  the  facilities  and  support  equipment  requirements  for 
each  site  from  user 

28  The  facilities  and  support  equipment  requirements  for 
each  site  are: 

29  Get  the  mission  types  from  user 

30  The  mission  types  are: 

31  Get  the  aircraft  configuration  for  each  mission  from  user 

32  The  aircraft  configuration  for  each  mission  is: 

33  Get  the  user  to  establish  mission  priorities 

34  The  mission  priorities  are: 

35  Get  user  to  determine  mission  delay  time  tolerances 
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36  The  tolerances  for  mission  delay  time  are: 

37  Coordinate  with  user  to  formulate  tasks  and/or  study 
measures  for  alternatives  being  examined 

38  The  mission  tasks  for  the  system  are: 

39  Get  aircraft  sortie  rates  from  user 

40  The  aircraft  sortie  rates  are: 

41  Get  user  to  determine  requirements  for  complementary 
missions 

42  The  requirements  for  complementary  missions  are: 

43  Review  the  lack  of  interface  data  with  user 

44  The  interfaces  with  other  systems  and  their  data  are: 

45  Get  data  for  interface  systems  from  user 

46  Get  number  of  aircraft  on  alert  at  each  location  from 
user 

47  The  number  of  aircraft  on  alert  at  each  location  is: 

48  Get  missions  to  be  flown  from  alert  information  from  user 

49  Missions  to  be  flown  from  alert  are: 

50  Get  frequency  of  alert  missions  information  from  user 

51  The  frequency  of  alert  missions  is: 

52  Get  alert  replacement  policy  information  from  user 

53  The  alert  replacement  policy  is: 

54  Get  user  to  determine  maintenance  concept  to  be  used 

55  The  maintenance  concept  to  be  used  is: 

56  Get  with  user  to  rectify  organization  structure  and 
maintenance  concept  level 

57  The  organization  structure  is: 

58  Get  with  user  to  rectify  differences  in  AFSC  structure, 
organization  structure,  and  maintenance  concept  level 

59  The  AFSC  structure  is: 
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Appendix  H:  COEA  Usage  Within  the  Acquisition  Lifecycle 


The  DoD  Acquisition  Management  System  has  five  phases: 
I.  Concept  Exploration  Definition,  II.  Demonstration  and 
Validation,  III.  Engineering  Manufacturing  Development, 

IV.  Production  and  Deployment,  and  V.  Operations  and  Support 
(O&S) . 

The  COEA  is  an  essential  part  of  the  DoD  acquisition  system. 
COEAs  are  required  for  DoD  Acquisition  Category  (ACAT)  I 
programs,  and  may  be  required  for  ACAT  II,  III,  and  IV 
programs . 


COEAs 

Pre-Milestone  0 :  Determination  of  Mission  Need 

The  COEA  process  should  begin  as  early  as  possible.  While 
there  is  no  specific  requirement  for  COEA  activities  prior 
to  milestone  0,  the  analysis  performed  to  identify  needs 
will  compare  the  threat,  current  capabilities,  and 
technology  opportunities  to  determine  whether  or  not  a  new 
development  effort  is  indicated. 


Phase  I :  Concept  Exploration  and  Definition 

Government  and  contractor  phase  I  studies  define  and  assess 
the  feasibility  and  rough  lifecycle  cost  estimates  of 
alternative  concepts  for  satisfying  the  identified  need. 
These  results  are  used  in  the  Phase  I  COEA  to  analyze  cost, 
schedule,  and  performance  tradeoffs  of  the  alternatives. 

.  The  phase  I  COEA:  (1)  identifies  the  advantages  and 
disadvantages  of  acquiring  a  new  system  over  modifying  the 
existing  one,  (2)  defines  the  characteristics  needed  in 
the  new  system  (i.e.,  performance  and  cost  goals  for  the 
next  phase) ,  and  (3)  screens  the  number  of  alternatives  to 
be  considered  in  later  phases. 


Phase  II:  Demonstration  and  Validation 

The  Phase  II  COEA  will  include  cost,  performance, 

supportability,  and  schedule  trade-offs  of  the  alternative 

concepts.  Cost  drivers  should  be  identified,  along  with 

maximum  cost  and  minimxim  performance  levels.  This  COEA  will 

be  more  detailed  than  the  Phase  I  COEA.  There  should  be  • 

fewer  and  more  clearly  defined  alternatives.  In  extreme 

cases,  concepts  discarded  at  milestone  I  may  be  reconsidered 

during  Phase  II.  ' 
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Phase  III:  Engineering  and  Manufacturing  Development 


The  Phase  III  COEA  may  be  only  an  update  of  the  Phase  II 
COEA.  However,  if  major  cost  or  performance  changes  have 
occurred  during  phase  II,  a  new  COEA  may  be  required.  The 
decision  authority  will  specify  the  elements  of  the  analysis 
that  require  updating. 

Phase  IV:  Production  and  Deployment 

If  a  major  revision  may  be  necessary,  the  decision  authority 
may  require  a  Phase  IV  COEA.  The  elements  of  this  analysis 
will  be  specified  as  part  of  the  planning  process. 


\ 


109 


Appendix  I:  Report  Results  —  COEA  Gatherer 


I .  Input  Data  From  Validation  Test 

The  scenario  is  a  _  format:  Wartime 

The  scenario/IOC  year  consistency  is:  Yes 

The  Engineering  Change  Proposal  is:  not  available 

The  number  of  locations  and  number  of  aircraft  at  each  site 
have:  not  been  determined 

The  supply  concept  (for  example : deploy  with  War  Readiness 
Spares  Kit  for  xxx  days)  has:  not  been  determined 

The  resupply  time  has:  not  been  determined 

The  extent  of  maintenance  capability  required  at  each  site 
has:  not  been  determined 

The  shelters  at  each  site  has:  not  been  determined 

The  facilities  and  support  equipment  for  each  site  have:  not 
been  determined 

The  mission  types  (for  example:  Interdiction;  Combat  Air 
Patrol)  have:  been  determined 

The  aircraft  configuration  for  each  mission  has:  been 
determined 

The  mission  priorities  have:  been  determined 

The  mission  cancellation  delay  times  tolerances  have:  been 
established 

The  mission  tasks  for  the  system  (based  on  need)  have:  been 
determined 

The  aircraft  sortie  rates  have:  been  determined 

The  requirements  for  complementary  missions  have:  been 
determined 

The  interfaces  with  other  systems  (for  example:  support 
aircraft  or  resources  for  particular  missions)  have:  been 
considered 

The  appropriate  interface  data  is:  available 
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The  number  of  aircraft  on  alert  at  each  location  is: 
available 

The  missions  to  be  flown  from  alert  have:  not  been 
determined 

The  frequency  of  alert  missions  has:  not  been  determined 

The  alert  replacement  policies  (for  example,  replacement 
when  launched  or  same  aircraft  return  from  alert)  have:  been 
determined 

The  maintenance  concept  (for  example:  2  level — remove  and 
replace;  3  level — repair  and  replace)  have:  not  been 
determined 

The  organizational  structure  and  maintenance  concept  level: 
matches 

The  AFSC  structure  and  organizational  structure  are:  in 
compliance 

The  AFSC  structure  and  maintenance  concept  level 
specifications  are:  in  compliance 

II.  Results  as  Displayed  bv  COEA  Gatherer 

The  following  is  available  for  LCCM  to  process  this  COEA: 

Scenario  is  Wartime 

The  mission  types  are: 

The  aircraft  configuration  for  each  mission  is: 

The  mission  priorities  are: 

The  tolerances  for  mission  delay  time  are: 

The  aircraft  sortie  rates  are: 

The  requirements  for  complementary  missions  are: 

The  number  of  aircraft  on  alert  at  each  location  is: 

The  alert  replacement  policy  is: 

The  organization  structure  is: 


The  AFSC  structure  is: 


♦ 

Th«  following  still  noads  to  ba  obtainad  for  LC^  or  issuas 
rasolvad: 

Check  validity  of  Engineering  Change  Proposal  with  user 

Get  the  numbers  and  locations  of  aircraft  at  each  site  from 
users 

Get  the  supply  concept  information  from  user 

Get  the  resupply  time  information  from  user 

Get  the  extent  of  maintenance  capability  required  at  each 
site  from  user 

Get  the  shelter  information  for  each  site  from  user 

Get  the  facilities  and  support  equipment  requirements  for 
each  site  from  user 

Get  missions  to  be  flown  from  alert  information  from  user 
Get  frequency  of  alert  missions  information  from  user 
Get  user  to  determine  maintenance  concept  to  be  used 

The  following  infornetion  la  available  for  TAC  THUNDBR  to 
process  this  COIA: 

Scenario  is  Wartime 

The  scenario/IOC  year  is: 

The  mission  types  are: 

The  aircraft  configuration  for  each  mission  is: 

The  mission  priorities  are: 

The  mission  tasks  for  the  system  are: 

The  aircraft  sortie  rates  are: 

The  requirements  for  complementary  missions  are: 

The  interfaces  with  other  systems  and  their  data  are: 

The  number  of  aircraft  on  alert  at  each  location  is: 

The  alert  replacement  policy  is: 
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Tha  following  information  still  needs  to  be  obtained  or 
problems  resolved  for  TAG  THUNDER: 

Get  the  nximbers  and  locations  of  aircraft  at  each  site  from 
risers 

Get  the  supply  concept  information  from  user 
Get  the  shelter  information  for  each  site  from  user 
Get  missions  to  be  flown  from  alert  information  from  user 
Get  frequency  of  alert  missions  information  from  user 
Get  user  to  determine  maintenance  concept  to  be  used 
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